From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 06:41:33 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 51B2EFDA; Sun, 10 Aug 2014 06:41:33 +0000 (UTC) Received: from mail-qc0-x22d.google.com (mail-qc0-x22d.google.com [IPv6:2607:f8b0:400d:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 063922D4E; Sun, 10 Aug 2014 06:41:32 +0000 (UTC) Received: by mail-qc0-f173.google.com with SMTP id w7so647188qcr.32 for ; Sat, 09 Aug 2014 23:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=pg6zNhqKtd7w3wYqd7Gp0NaPTyTmjwbvfnlf0qUhEkA=; b=XfL+61wnKbZ+CvXPddbH2PG/UXVga4X6VRFnc6ji/B6vuXibt5onQS5Kgwj7s67pyW EduIiI8i+97tjJjVGiQG8V5P0d73a1taOtUxoiCgB6bXj6ulkmlhCvm8SUdDsk7RCg5H SXiDpM8urSlEH13NOBWNQlmKhQMuJRlGYx+W/0mZ6YdDgEz0tAL2OFeP9INK5m5WpvB6 96GYXeEWWs6dzLye+MQhLddoma6iJ2bGKD8UbN75pwmOAb9dTnAlZ3vtL99D3TboZHpq o3vThGegGICH/d2SbNEU2Hn+/0mgWXYNOo4ADAdsLfIUsaLNQu2oPTadriRUOZfl2+sK BzoQ== MIME-Version: 1.0 X-Received: by 10.140.41.133 with SMTP id z5mr36483236qgz.99.1407652892080; Sat, 09 Aug 2014 23:41:32 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.41.6 with HTTP; Sat, 9 Aug 2014 23:41:32 -0700 (PDT) Date: Sat, 9 Aug 2014 23:41:32 -0700 X-Google-Sender-Auth: LbUeRuk5SmZyiL3t9SIeH9yWlu0 Message-ID: Subject: [rfc] INJECT mode for net80211 From: Adrian Chadd To: "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 06:41:33 -0000 Hi! I kinda got fed up with the lack of functioning inject. * monitor mode isn't inject mode; * ahdemo mode seems .. less useful. So I just created IEEE80211_M_INJECT and taught net80211 / ath about it. This is like monitor mode (straight to RUN, no need to set an SSID, no auto scanning by default) but it allows transmit and populates the node table with temporary node entries. So: http://people.freebsd.org/~adrian/ath/20140809-net80211-ath-inject-1.diff I'd like to commit this in the next couple of days. I've also tested this with aircrack-ng - the built port didn't work with monitor mode modified to inject frames. I don't know why. Rebuilt from source worked fine. I've just modified my local copy to set the IFM_IEEE80211_INJECT flag rather than monitor flag and it seems it's all okay. The only issue I've seen with packet injection is that aircrack-ng isn't waiting until the interface is up before trying to send frames, so some are rejected rather than buffered. -a From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 07:54:12 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90D9BAB; Sun, 10 Aug 2014 07:54:12 +0000 (UTC) Received: from felyko.com (felyko.com [65.49.80.26]) by mx1.freebsd.org (Postfix) with ESMTP id 7AF9E24AB; Sun, 10 Aug 2014 07:54:12 +0000 (UTC) Received: from [IPv6:2601:9:8280:5fd:d5a4:20d0:8128:6903] (unknown [IPv6:2601:9:8280:5fd:d5a4:20d0:8128:6903]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by felyko.com (Postfix) with ESMTPSA id 7806334AAC9; Sun, 10 Aug 2014 00:54:11 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [rfc] INJECT mode for net80211 From: Rui Paulo In-Reply-To: Date: Sun, 10 Aug 2014 00:54:26 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> References: To: Adrian Chadd X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 07:54:12 -0000 On Aug 9, 2014, at 23:41, Adrian Chadd wrote: > Hi! >=20 > I kinda got fed up with the lack of functioning inject. >=20 > * monitor mode isn't inject mode; > * ahdemo mode seems .. less useful. >=20 > So I just created IEEE80211_M_INJECT and taught net80211 / ath about > it. This is like monitor mode (straight to RUN, no need to set an > SSID, no auto scanning by default) but it allows transmit and > populates the node table with temporary node entries. >=20 > So: >=20 > = http://people.freebsd.org/~adrian/ath/20140809-net80211-ath-inject-1.diff This patch looks incomplete. Did you forget to diff sys/net? > I'd like to commit this in the next couple of days. >=20 > I've also tested this with aircrack-ng - the built port didn't work > with monitor mode modified to inject frames. I don't know why. Rebuilt > from source worked fine. I've just modified my local copy to set the > IFM_IEEE80211_INJECT flag rather than monitor flag and it seems it's > all okay. >=20 > The only issue I've seen with packet injection is that aircrack-ng > isn't waiting until the interface is up before trying to send frames, > so some are rejected rather than buffered. In general, I'd prefer to have one mode. aircrack used to work in = monitor mode, so I'm surprised the problem is the lack of an inject = mode. Looking at the code, you're pretty much defining INJECT to be = MONITOR mode. -- Rui Paulo From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 08:02:55 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 78507455; Sun, 10 Aug 2014 08:02:55 +0000 (UTC) Received: from mail-qg0-x236.google.com (mail-qg0-x236.google.com [IPv6:2607:f8b0:400d:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 183A2256A; Sun, 10 Aug 2014 08:02:55 +0000 (UTC) Received: by mail-qg0-f54.google.com with SMTP id z60so7533257qgd.41 for ; Sun, 10 Aug 2014 01:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=N2pVFlTcrQI7sHhImRcFlsYCPWOLJq8VjYaZdzFmrio=; b=b1jGoQ21Fz/7cDf49psjp1uWJiCBG/IyATp2VgrLrC/6A/KIgNxel5q+Gn1gH3nJxs cEqYXfS1Ea/HJKKT4+ktzLF3+bsiO6jRWqMts6W8sr2UDC3BzbQuZGzSiV8OfI/8wlmR gk/7ND1dAYMNnWKexqvG7RK2wu55XIodlarl+nqkm3H1M0eoqU7IA9+UFOi/KT4+Sfx4 F5B5DEOn1ixqy7Dx31CPxoUW6IZ6MMb653Uj1eQOLkzl7uuAc54DCjFx9kClMZ9EUK/v hMhquolIOdwhwuUkROGFFAR5f4UQvPjm3q08Zhsxlnl+RbHG+0X2MIycDkXdn6yvjuOr wVoA== MIME-Version: 1.0 X-Received: by 10.229.73.70 with SMTP id p6mr52293368qcj.13.1407657774195; Sun, 10 Aug 2014 01:02:54 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.41.6 with HTTP; Sun, 10 Aug 2014 01:02:54 -0700 (PDT) In-Reply-To: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> References: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> Date: Sun, 10 Aug 2014 01:02:54 -0700 X-Google-Sender-Auth: Kq-FoBV3ymBFn6OJH7SKLETfNqY Message-ID: Subject: Re: [rfc] INJECT mode for net80211 From: Adrian Chadd To: Rui Paulo Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 08:02:55 -0000 On 10 August 2014 00:54, Rui Paulo wrote: > On Aug 9, 2014, at 23:41, Adrian Chadd wrote: > >> Hi! >> >> I kinda got fed up with the lack of functioning inject. >> >> * monitor mode isn't inject mode; >> * ahdemo mode seems .. less useful. >> >> So I just created IEEE80211_M_INJECT and taught net80211 / ath about >> it. This is like monitor mode (straight to RUN, no need to set an >> SSID, no auto scanning by default) but it allows transmit and >> populates the node table with temporary node entries. >> >> So: >> >> http://people.freebsd.org/~adrian/ath/20140809-net80211-ath-inject-1.diff > > This patch looks incomplete. Did you forget to diff sys/net? Hm, try: http://people.freebsd.org/~adrian/ath/20140809-net80211-ath-inject-2.diff > In general, I'd prefer to have one mode. aircrack used to work in monitor mode, so I'm surprised the problem is the lack of an inject mode. Looking at the code, you're pretty much defining INJECT to be MONITOR mode. How did it used to work? * monitor mode very specifically doesn't define output methods at all; * there's no temporary nodes created when transmitting, so it all simply ends up being dropped because no txnode was found. So perhaps there was some path via the raw output method which no longer is working. The _output() method supposedly permits BPF injected packets to work by sending packets using a node == vap->iv_bss, but that can't possibly work with the existing monitor mode because code in ieee80211.c doesn't set the output methods for monitor mode. -a From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 08:19:43 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 059B2761; Sun, 10 Aug 2014 08:19:43 +0000 (UTC) Received: from mail-qa0-x22f.google.com (mail-qa0-x22f.google.com [IPv6:2607:f8b0:400d:c00::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 993B3266A; Sun, 10 Aug 2014 08:19:42 +0000 (UTC) Received: by mail-qa0-f47.google.com with SMTP id i13so6966297qae.6 for ; Sun, 10 Aug 2014 01:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=I9nybPXIqZ4l2OuA9vcjY4RjTg/wuH9WpXNdxVXQHPs=; b=ne+HAbfeMrcAplYdomCiEO6AshinECM5BFM/nhvN3NJb5CzaKo8Q+mkfffkACE8yvn 2YC9d0tgs9maXwMn70YXLkscCPI0D2LiuaFUVTPtmadgbVKo9gc0hR82lJpLhCkEW4KE kDP8MQf1B6H5+5D/z/YcSvVROQ9V44C4m1j2pJzkcSXqp2Dh+egUb5rc6Bf3+e6lIrwf 1e/u0sKcVQioiUzLygHTnngP3reDi1+2e90lH3GxT6TOimqNNM0WWu+0n0DseVWbXNA/ GT77KS2jCVJHx/cfcFeo8FnBA3XA0vSGEnwxJYwWPPpMHtinSlk0ScWwowLT7JaXcRCg KykQ== MIME-Version: 1.0 X-Received: by 10.224.120.138 with SMTP id d10mr54303961qar.55.1407658781766; Sun, 10 Aug 2014 01:19:41 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.41.6 with HTTP; Sun, 10 Aug 2014 01:19:41 -0700 (PDT) In-Reply-To: References: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> Date: Sun, 10 Aug 2014 01:19:41 -0700 X-Google-Sender-Auth: swgpTu3LF9mwFkNO4dKolbB9h7A Message-ID: Subject: Re: [rfc] INJECT mode for net80211 From: Adrian Chadd To: Rui Paulo Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 08:19:43 -0000 ah, sam disabled it in r195846 due to regulatory constraints. Hm. I wonder what else rotted in the meantime. -a From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 08:28:23 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F02F6A39; Sun, 10 Aug 2014 08:28:23 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8EEA02728; Sun, 10 Aug 2014 08:28:23 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so7484988qgd.40 for ; Sun, 10 Aug 2014 01:28:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=iOLHBPLcTXTPcreGRJDx1CRtqevSFZPek5luWTmk+JU=; b=NzGZHdHmL7ruoxgqdEjZvwV+iFMz6nhxw6/LWGikGpCoGAOLWaVXFL4k/fb43luMVI 3cKSSWk9w9qoU7c0tO+E2sG7gYfvAvNOBPOuwKk4dGpqB7m1Zn3Cvb8i4BUTsiPrSWsU F9IxvzFZ+pOTLOtm9nHHIawAcfAyWhiMlbSq5+doZQSGml+ze29DBUfgTDLFC0eHCIe3 u/wVw6++XhPvjGfpVygH4riOBz08DawXvlltotR6sQ9Q9//bV32N/bIkrd/sXisL0XmZ PCcbMmfqFhJ065wx/kcjwDYokmrTTY90T62pEyF59bBe1CqCddM9PMhmUGeiYcOK4j/q Fkyw== MIME-Version: 1.0 X-Received: by 10.229.38.3 with SMTP id z3mr52303939qcd.17.1407659302757; Sun, 10 Aug 2014 01:28:22 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.41.6 with HTTP; Sun, 10 Aug 2014 01:28:22 -0700 (PDT) In-Reply-To: References: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> Date: Sun, 10 Aug 2014 01:28:22 -0700 X-Google-Sender-Auth: O2Eex4biiv7KYNlpYvWhjPpDxXE Message-ID: Subject: Re: [rfc] INJECT mode for net80211 From: Adrian Chadd To: Rui Paulo Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 08:28:24 -0000 ... interesting. Ok, so: * if I just "remove" sam's patch there, monitor mode works to inject; * I've patched kismet so it reads the radiotap data using the example parser and added XCHANNEL support - so now it actually works nicely; * aircrack-ng from ports doensn't send - it's triggering on this in net80211: wh = mtod(m, struct ieee80211_frame *); if ((wh->i_fc[0] & IEEE80211_FC0_VERSION_MASK) != IEEE80211_FC0_VERSION_0) senderr(EIO); /* XXX */ .. I haven't looked into why yet. * but, aircrack-ng built from source works, save for when it tries to transmit too quickly after changing channels. Ok, so I'm going to just revert that change for now and see about figuring out some other way to enforce regulatory concerns on monitor mode transmit. (Likely by allowing receive, but failing transmit on non-regulatory channels.) I'll speak to the Kismet author here tomorrow and see about getting this replacement radiotap parser in so it works correctly again in FreeBSD. And as for aircrack-ng - guess I'll poke the port maintainer. -a From owner-freebsd-arch@FreeBSD.ORG Sun Aug 10 08:42:15 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E07791F1; Sun, 10 Aug 2014 08:42:14 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95110289D; Sun, 10 Aug 2014 08:42:14 +0000 (UTC) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 315536A6005; Sun, 10 Aug 2014 10:42:11 +0200 (CEST) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.7/8.14.7) with ESMTP id s7A8gBj8050615; Sun, 10 Aug 2014 10:42:11 +0200 (CEST) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.7/8.14.7/Submit) id s7A8gA76050233; Sun, 10 Aug 2014 10:42:10 +0200 (CEST) (envelope-from lars) Date: Sun, 10 Aug 2014 10:42:10 +0200 From: Lars Engels To: Adrian Chadd Subject: Re: [rfc] INJECT mode for net80211 Message-ID: <20140810084210.GA56043@e-new.0x20.net> References: <80DB3DCF-496E-4596-B9F3-4D86AC82DEC5@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="LQksG6bCIzRHxTLp" Content-Disposition: inline In-Reply-To: X-Editor: VIM - Vi IMproved 7.4 X-Operation-System: FreeBSD 8.4-RELEASE-p4 User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-wireless@freebsd.org" , Rui Paulo , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Aug 2014 08:42:15 -0000 --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Aug 10, 2014 at 01:28:22AM -0700, Adrian Chadd wrote: > ... interesting. Ok, so: >=20 > * if I just "remove" sam's patch there, monitor mode works to inject; > * I've patched kismet so it reads the radiotap data using the example > parser and added XCHANNEL support - so now it actually works nicely; > * aircrack-ng from ports doensn't send - it's triggering on this in net80= 211: >=20 > wh =3D mtod(m, struct ieee80211_frame *); > if ((wh->i_fc[0] & IEEE80211_FC0_VERSION_MASK) !=3D > IEEE80211_FC0_VERSION_0) > senderr(EIO); /* XXX */ >=20 > .. I haven't looked into why yet. >=20 > * but, aircrack-ng built from source works, save for when it tries to > transmit too quickly after changing channels. >=20 > Ok, so I'm going to just revert that change for now and see about > figuring out some other way to enforce regulatory concerns on monitor > mode transmit. (Likely by allowing receive, but failing transmit on > non-regulatory channels.) >=20 > I'll speak to the Kismet author here tomorrow and see about getting > this replacement radiotap parser in so it works correctly again in > FreeBSD. >=20 > And as for aircrack-ng - guess I'll poke the port maintainer. >=20 Which is me. :) Thanks a lot for working on this, let me know if you need any assistance for the aircrack-ng port. BTW the aircrack-ng developers are very helpful and FreeBSD-friendly, so we should get patches upstream. --LQksG6bCIzRHxTLp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJT5zBiXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RjQwMDE3RTRERjUzMTI1N0FGRTUxNDlF NTRDQjM3RDNBMDg5RDZEAAoJEOVMs306CJ1tPCoIAK8aQcJYSRSwIFlx5mahl7Lq lR3saWlYyhOTeCVOjwKAoQJiV1z7NbKYRlEPC9nCiC892T62pmcW47YLnTla70Rq EAJ3meeUnGxopSAIfq9VUF9K6Sc92XU99FDHD8k4FA6FokSPJ8MSgYrR6oUxCwyE iQW+GW5u5R0Qo0Ltl7Mx12xWr4OWd7yDSJ9u8DP25aK3xonvQc6aPN465+dtoxAJ GKm89ezD548vdOCqUHqaARr23Xos7R3+ZNWsNo9A3yl4fhbio7sP5MwAUCBQw7eW j9jEIAqnbJiaCvsghv3hDT2GQlFbjM4Xo7V9IZvPk3NFjE99PnZDcmge/DRbKmY= =E3Af -----END PGP SIGNATURE----- --LQksG6bCIzRHxTLp-- From owner-freebsd-arch@FreeBSD.ORG Mon Aug 11 21:28:53 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 71F48262 for ; Mon, 11 Aug 2014 21:28:53 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48E162705 for ; Mon, 11 Aug 2014 21:28:53 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C1A55B924; Mon, 11 Aug 2014 17:28:51 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: current fd allocation idiom Date: Mon, 11 Aug 2014 11:24:51 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140717235538.GA15714@dft-labs.eu> <20140718155959.GN93733@kib.kiev.ua> <20140718191928.GB7179@dft-labs.eu> In-Reply-To: <20140718191928.GB7179@dft-labs.eu> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408111124.52064.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 11 Aug 2014 17:28:51 -0400 (EDT) Cc: Konstantin Belousov , Mateusz Guzik X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Aug 2014 21:28:53 -0000 On Friday, July 18, 2014 3:19:28 pm Mateusz Guzik wrote: > On Fri, Jul 18, 2014 at 06:59:59PM +0300, Konstantin Belousov wrote: > > On Fri, Jul 18, 2014 at 04:40:12PM +0200, Mateusz Guzik wrote: > > > On Fri, Jul 18, 2014 at 04:06:29PM +0300, Konstantin Belousov wrote: > > > > It seems that all what is needed is conversion of places using > > > > falloc() to falloc_noinstall()/finstall(). > > > > > > > > > > This postpones fd allocation to after interested function did all work > > > it wanted to do, which means we would need reliable ways of reverting > > > all the work in case allocation failed. I'm not so confident we can do > > > that for all current consumers and both current and my proposed approach > > > don't impose such requirement. > > Cleanup should be identical to the actions done on close(2). > > > > > > > > Of course postponing fd allocation where possible is definitely worth > > > doing. > > Yes, and after that the rest of the cases should be evaluated. > > But my gut feeling is that everything would be converted. > > > > So let's say you accept() a connection. > > With current code, if you got to accept the connection you got it. > > With your proposal you may find that you can't allocate any fd and have > to close fp. This will be visible as accept + close by the other end, > while the caller never saw the connection. > > My guess is people would complain once they encounter such issue. Can't you already get this if you overflow the listen queue? (Having "accepted" connections aborted where the user application doesn't know): } else { /* * Keep removing sockets from the head until there's room for * us to insert on the tail. In pre-locking revisions, this * was a simple if(), but as we could be racing with other * threads and soabort() requires dropping locks, we must * loop waiting for the condition to be true. */ while (head->so_incqlen > head->so_qlimit) { struct socket *sp; sp = TAILQ_FIRST(&head->so_incomp); TAILQ_REMOVE(&head->so_incomp, sp, so_list); head->so_incqlen--; sp->so_qstate &= ~SQ_INCOMP; sp->so_head = NULL; ACCEPT_UNLOCK(); soabort(sp); ACCEPT_LOCK(); } TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list); so->so_qstate |= SQ_INCOMP; head->so_incqlen++; I think the simplest approach would be to first convert as many places as possible to use falloc_noinstall() / finstall(). If you end up with all of them converted then you can just rename falloc_noinstall to falloc() and retire the old falloc(). -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Mon Aug 11 21:28:54 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB44E263; Mon, 11 Aug 2014 21:28:54 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2BB32706; Mon, 11 Aug 2014 21:28:54 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 9F89EB948; Mon, 11 Aug 2014 17:28:53 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Mon, 11 Aug 2014 11:54:17 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140730053446.DCE8D580A2@chaos.jnpr.net> <53D944F5.7000207@freebsd.org> In-Reply-To: <53D944F5.7000207@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408111154.17357.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 11 Aug 2014 17:28:53 -0400 (EDT) Cc: Alfred Perlstein , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Aug 2014 21:28:54 -0000 On Wednesday, July 30, 2014 3:18:13 pm Alfred Perlstein wrote: > The goal of a GSOC project is to get the code into FreeBSD. Just to respond to this point: not necessarily. I would say the primary goal is to build relationships with the students and recruit new developers. If we get useful code out of the process as well, that's great, but it isn't the priamry goal. In addition, doing a GSOC project isn't a guarantee of getting code into the tree (even though it may often work out that way), and we should avoid marketing it as such. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Mon Aug 11 21:28:56 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08067265 for ; Mon, 11 Aug 2014 21:28:56 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2AB42707 for ; Mon, 11 Aug 2014 21:28:55 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A8C54B94A; Mon, 11 Aug 2014 17:28:54 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: RFC: cpuid_t Date: Mon, 11 Aug 2014 12:18:29 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140810060747.V855@besplex.bde.org> In-Reply-To: <20140810060747.V855@besplex.bde.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408111218.29802.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 11 Aug 2014 17:28:54 -0400 (EDT) Cc: Adrian Chadd X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Aug 2014 21:28:56 -0000 On Saturday, August 09, 2014 5:16:26 pm Bruce Evans wrote: > On Sat, 9 Aug 2014, Adrian Chadd wrote: > > > How's this look? > > Not good. > > > Index: sys/sys/_types.h > > =================================================================== > > --- sys/sys/_types.h (revision 269480) > > +++ sys/sys/_types.h (working copy) > > @@ -52,6 +52,7 @@ > > typedef __uint16_t __nlink_t; /* link count */ > > typedef __int64_t __off_t; /* file offset */ > > typedef __int32_t __pid_t; /* process [group] */ > > +typedef __uint32_t __cpuid_t; /* CPU ID */ > > typedef __int64_t __rlim_t; /* resource limit - intentionally */ > > /* signed, because of legacy code */ > > /* that uses -1 for RLIM_INFINITY */ > > Unsorting. In the English alphabet, c is not between p and r. > > The whitespace seems to be inconsistent, but the mail corrupted all the > tabs so it is hard to tell. > > > Index: sys/sys/types.h > > =================================================================== > > --- sys/sys/types.h (revision 269480) > > +++ sys/sys/types.h (working copy) > > @@ -154,6 +154,11 @@ > > #define _LWPID_T_DECLARED > > #endif > > > > +#ifndef _CPUID_T_DECLARED > > +typedef __cpuid_t cpuid_t; /* CPU ID */ > > +#define _CPUID_T_DECLARED > > +#endif > > + > > #ifndef _MODE_T_DECLARED > > typedef __mode_t mode_t; /* permissions */ > > #define _MODE_T_DECLARED > > Unsorting. c is also not between l and m. > > The comment is banal, like many nearby ones. It does less than echo the > code. Most for ids at least echo the code by repeating "id" in lower case. > > In private mail, I said that this typedef is less needed than one for > file descriptors. Since the latter need is negative, the need for this > one is very negative. Ids should be small integers so that they can be > used directly as array indexes, unless they need to cover a large sparse > namespace so that they need to be hash numbers. You would just prefer an int then? Your point about it being used in kinfo is well-taken (meaning that once it is part of an ABI you can't actually change it transparently). I know Adrian is currently concerned about doing the initial sweep to identify the places that store CPU IDs and marking them as such, and I believe he sees value in simply enumerating the places that are CPU IDs rather than something else. It is true that cpuset_t currently assumes it represents a set of integers (more or less) and the closes analogy to that (signals) just use plain int for signal numbers (and not a sig_t or the like). -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Aug 12 00:31:43 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8128D5AB; Tue, 12 Aug 2014 00:31:43 +0000 (UTC) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3347C2A05; Tue, 12 Aug 2014 00:31:43 +0000 (UTC) Received: by mail-qc0-f174.google.com with SMTP id l6so2266356qcy.33 for ; Mon, 11 Aug 2014 17:31:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=8DrnTpjHGm/cHKMbtCOtfvzGhI9Y1L/FlwlXvTPiFMk=; b=YWDiTQptD4VHwafvzc0HOKA/GIMYVg30bX3QgNK6+Df5xq0nEah9rG3LwTyJVwvqPg Yn8nnc5WxgYDBgVRIyZM9qGrOknGfE2lsKjPYy7lLJ27pMpOElwVYZFLlmC2dgw+1Bji VuCjZqIXa9tfkVij7Bm1k5o4X84vnxU+jERAah39utdaEj6IFIUXhznyMEoRIP5+asLx AzzvS1i50gw34Suup4c0VRAGyq1bX/TbwT2VduVkI4AeOSQz2+3VTkJtLSyzxiXyy7Kb j2YztGavPYoCvSdhf5PjYdw1aWMEdauDSPVQ9Hssq5SLEd3KejyAFhBwQTUT92WLCu/o qRYA== MIME-Version: 1.0 X-Received: by 10.224.15.195 with SMTP id l3mr1286389qaa.98.1407803502424; Mon, 11 Aug 2014 17:31:42 -0700 (PDT) Received: by 10.224.41.6 with HTTP; Mon, 11 Aug 2014 17:31:42 -0700 (PDT) In-Reply-To: <201408111218.29802.jhb@freebsd.org> References: <20140810060747.V855@besplex.bde.org> <201408111218.29802.jhb@freebsd.org> Date: Mon, 11 Aug 2014 17:31:42 -0700 Message-ID: Subject: Re: RFC: cpuid_t From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 00:31:43 -0000 On 11 August 2014 09:18, John Baldwin wrote: > You would just prefer an int then? Your point about it being used in kinfo > is well-taken (meaning that once it is part of an ABI you can't actually > change it transparently). I know Adrian is currently concerned about doing > the initial sweep to identify the places that store CPU IDs and marking them > as such, and I believe he sees value in simply enumerating the places that > are CPU IDs rather than something else. It is true that cpuset_t currently > assumes it represents a set of integers (more or less) and the closes analogy > to that (signals) just use plain int for signal numbers (and not a sig_t or > the like). Right. Right now I think we're stuck in the really stupid position that: * > 253 "schedulable entity" machines exist right now (think all that silly sparc hardware that exists); * and > 253 core Intel/AMD machines are going to show up soon; * along with that weird MIPS project that Robert and co are working on; * .. but I'm trying to tackle this as a "fun, spare time project." So my approach is designed to make it easy to get some changes into the tree without waiting for "final" thing to show up before commiting it. Ie: * a char / u_char / something variable and non-defined is going to bite us during the FreeBSD-11 timeframe; * so I'd rather spend my time enumerating all the places a cpu identifier is used and, well, change it to cpuid_t without changing anything else about the behaviour; * .. and fix whatever basic API things show up because of that - john and bruce have both shown me some of those; * .. then stop at that - (eg, I'm not going to try and fix all the places where cpuid_t is used as some "thing" to iterate from/to) - that makes iterating over a sparse set of CPU IDs rather .. amusing; * then at some later stage we can hammer out a more stable API - likely as part of FreeBSD-12. I'm under no real expectation that we're going to get this _right_ given the FreeBSD-11 timeframe and that I'm doing this as a spare time project. However, i want to get at least the more dumb dumbness out of the tree before we branch 11 so if we do have to change anything to do with userland ABI, it'll hurt "less". I'd like to be able to use grep to figure out where all the users of a cpu id are so we have a reasonably tractable place to start in fixing things, as well as laying down some basic API rules for the future (eg, no new function calls that manipulate something about a cpu_id should be passing in a char or u_char - and this did just happen recently.) So, I'd like to focus more on getting the typedef into there and in the right place - so thankyou Bruce, I do appreciate the feedback and I'm going to fix things before I submit this again. -a From owner-freebsd-arch@FreeBSD.ORG Tue Aug 12 19:37:06 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6A1FD9E for ; Tue, 12 Aug 2014 19:37:06 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B23125C8 for ; Tue, 12 Aug 2014 19:37:06 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 141CBB93A; Tue, 12 Aug 2014 15:37:05 -0400 (EDT) From: John Baldwin To: Adrian Chadd Subject: Re: RFC: cpuid_t Date: Tue, 12 Aug 2014 09:39:56 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <201408111218.29802.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201408120939.56176.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 12 Aug 2014 15:37:05 -0400 (EDT) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 19:37:07 -0000 On Monday, August 11, 2014 8:31:42 pm Adrian Chadd wrote: > On 11 August 2014 09:18, John Baldwin wrote: > > > You would just prefer an int then? Your point about it being used in kinfo > > is well-taken (meaning that once it is part of an ABI you can't actually > > change it transparently). I know Adrian is currently concerned about doing > > the initial sweep to identify the places that store CPU IDs and marking them > > as such, and I believe he sees value in simply enumerating the places that > > are CPU IDs rather than something else. It is true that cpuset_t currently > > assumes it represents a set of integers (more or less) and the closes analogy > > to that (signals) just use plain int for signal numbers (and not a sig_t or > > the like). > > Right. > > Right now I think we're stuck in the really stupid position that: > > * > 253 "schedulable entity" machines exist right now (think all that > silly sparc hardware that exists); > * and > 253 core Intel/AMD machines are going to show up soon; > * along with that weird MIPS project that Robert and co are working on; > * .. but I'm trying to tackle this as a "fun, spare time project." > > So my approach is designed to make it easy to get some changes into > the tree without waiting for "final" thing to show up before commiting > it. > > Ie: > > * a char / u_char / something variable and non-defined is going to > bite us during the FreeBSD-11 timeframe; > * so I'd rather spend my time enumerating all the places a cpu > identifier is used and, well, change it to cpuid_t without changing > anything else about the behaviour; > * .. and fix whatever basic API things show up because of that - john > and bruce have both shown me some of those; > * .. then stop at that - (eg, I'm not going to try and fix all the > places where cpuid_t is used as some "thing" to iterate from/to) - > that makes iterating over a sparse set of CPU IDs rather .. amusing; > * then at some later stage we can hammer out a more stable API - > likely as part of FreeBSD-12. > > I'm under no real expectation that we're going to get this _right_ > given the FreeBSD-11 timeframe and that I'm doing this as a spare time > project. > > However, i want to get at least the more dumb dumbness out of the tree > before we branch 11 so if we do have to change anything to do with > userland ABI, it'll hurt "less". I'd like to be able to use grep to > figure out where all the users of a cpu id are so we have a reasonably > tractable place to start in fixing things, as well as laying down some > basic API rules for the future (eg, no new function calls that > manipulate something about a cpu_id should be passing in a char or > u_char - and this did just happen recently.) > > So, I'd like to focus more on getting the typedef into there and in > the right place - so thankyou Bruce, I do appreciate the feedback and > I'm going to fix things before I submit this again. I think Bruce's point is you can just fix the few places that use a u_char to an int and call it a day. The biggest offender of those that I can think of are td_oncpu/td_lastcpu and the corresponding fields in kinfo along with expanding NOCPU to be -1 instead of 255. I think most other places already use an int (pf is one place that doesn't, it borrows N bits from some other field to store the CPU number, so it can't use a cpuid_t anyway), so the patch to just use an int might actually be quite small. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Aug 12 22:17:28 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 636D75A6; Tue, 12 Aug 2014 22:17:28 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 14F0B2B71; Tue, 12 Aug 2014 22:17:28 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so10346317qgd.12 for ; Tue, 12 Aug 2014 15:17:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xWx4J7sWB2Crk2FDbMpJNoj3oaKBIxDd4x7l5sZuINs=; b=PIld2/eh0LTMwiYpqGJvSMHhZbZrpx95yV35k2ZHAGIfdYS6MmG5Grdw67uiRrgNC5 ccrlVpvALx/4ixYFG6peEt2m91Ga4+EXlPzvvdrEmIcQHz8oJFEyh4B1U435WijOiXXY q/TvdleH86wSlYKaFGoduOTKSdcaIc7lW2PiZ+qQhPzEcKBQotcMBggZZOO2cOkfHIuP 1Iiou6ZmMLWdYF1cLlkjV2NZBBnm6Mb0atdbhSNsZU2RS25anVW2yOgl7+ajqmLh+uqc iNwA0CEMZspHcuGwAmrTGy82axrHW4mL7wQozTB4QzbuzD0oPf8rD5x11pbF4M654kOu cWkA== MIME-Version: 1.0 X-Received: by 10.140.19.201 with SMTP id 67mr988720qgh.28.1407881847078; Tue, 12 Aug 2014 15:17:27 -0700 (PDT) Received: by 10.224.39.139 with HTTP; Tue, 12 Aug 2014 15:17:27 -0700 (PDT) In-Reply-To: <201408120939.56176.jhb@freebsd.org> References: <201408111218.29802.jhb@freebsd.org> <201408120939.56176.jhb@freebsd.org> Date: Tue, 12 Aug 2014 15:17:27 -0700 Message-ID: Subject: Re: RFC: cpuid_t From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 22:17:28 -0000 On 12 August 2014 06:39, John Baldwin wrote: > I think Bruce's point is you can just fix the few places that use a u_char to > an int and call it a day. The biggest offender of those that I can think of > are td_oncpu/td_lastcpu and the corresponding fields in kinfo along with > expanding NOCPU to be -1 instead of 255. I think most other places already > use an int (pf is one place that doesn't, it borrows N bits from some other > field to store the CPU number, so it can't use a cpuid_t anyway), so the patch > to just use an int might actually be quite small. You could do that, but then we'd be left in a situation that it isn't 100% clear what type people should be using (except hoping they cut/paste code that uses an int and don't make dumb decisions like assuming they can pack it into other fields, like pf) and that they could treat the CPU ID as some kind of index number. Being able to grep for all the places where cpuid_t is will make it a lot easier in the future to uplift all of the cpu id manipulating code to treat CPU IDs differently (eg making things very sparse.) So, I'm happy if the group decision is to just make it an int. It just has to be done before FreeBSD-11 ships. :) I don't have such an aversion to typedefs. It has always made it much easier to do type checking and changes to things in C that I'd normally do in C++ with classes. -a From owner-freebsd-arch@FreeBSD.ORG Tue Aug 12 23:36:24 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 347CF86B; Tue, 12 Aug 2014 23:36:24 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A6E526BB; Tue, 12 Aug 2014 23:36:23 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id m15so10551940wgh.29 for ; Tue, 12 Aug 2014 16:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Rqg7qTNYQktDwbS7h++60UQQhzncoH+D9BkJ3RUGK7g=; b=R9GWX5Qjpl2JOoyYbmYDpgajq3FlKkokHaiC1VTEmeH5l4OrAGX5SmeMAnXgm6c61t 7fKzK9DpGz8IDQeqzgh48vxNmhGrra1XV43AFSJACWbaZbqNSCMyni1Z+k4bD3g0cW/s HNaQdW96HCJOWxusKdrhBqdr1oogHTcDXXLx0toHWp/6mVPiQtcJBhBc1nATIQZC5eIk noim19NP6r2vT0LUhne7ySq2o5tVVeb1eLBAS+2L3xIsMHvnHFs6dbGIShDkdt8+ubLD 6f4F1wJH55cLFCjc8WLAcN++uUbuQexoIr6VeOPgZ+NwFm4c133N9T/zBA2WAUZmjFuR 8jzA== X-Received: by 10.194.110.7 with SMTP id hw7mr863796wjb.38.1407886581762; Tue, 12 Aug 2014 16:36:21 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id ex2sm395918wjd.30.2014.08.12.16.36.20 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Tue, 12 Aug 2014 16:36:20 -0700 (PDT) Date: Wed, 13 Aug 2014 01:36:18 +0200 From: Mateusz Guzik To: John Baldwin Subject: Re: current fd allocation idiom Message-ID: <20140812233617.GA17869@dft-labs.eu> References: <20140717235538.GA15714@dft-labs.eu> <20140718155959.GN93733@kib.kiev.ua> <20140718191928.GB7179@dft-labs.eu> <201408111124.52064.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201408111124.52064.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 23:36:24 -0000 On Mon, Aug 11, 2014 at 11:24:51AM -0400, John Baldwin wrote: > On Friday, July 18, 2014 3:19:28 pm Mateusz Guzik wrote: > > On Fri, Jul 18, 2014 at 06:59:59PM +0300, Konstantin Belousov wrote: > > > On Fri, Jul 18, 2014 at 04:40:12PM +0200, Mateusz Guzik wrote: > > > > On Fri, Jul 18, 2014 at 04:06:29PM +0300, Konstantin Belousov wrote: > > > > > It seems that all what is needed is conversion of places using > > > > > falloc() to falloc_noinstall()/finstall(). > > > > > > > > > > > > > This postpones fd allocation to after interested function did all work > > > > it wanted to do, which means we would need reliable ways of reverting > > > > all the work in case allocation failed. I'm not so confident we can do > > > > that for all current consumers and both current and my proposed approach > > > > don't impose such requirement. > > > Cleanup should be identical to the actions done on close(2). > > > > > > > > > > > Of course postponing fd allocation where possible is definitely worth > > > > doing. > > > Yes, and after that the rest of the cases should be evaluated. > > > But my gut feeling is that everything would be converted. > > > > > > > So let's say you accept() a connection. > > > > With current code, if you got to accept the connection you got it. > > > > With your proposal you may find that you can't allocate any fd and have > > to close fp. This will be visible as accept + close by the other end, > > while the caller never saw the connection. > > > > My guess is people would complain once they encounter such issue. > > Can't you already get this if you overflow the listen queue? (Having > "accepted" connections aborted where the user application doesn't know): > > } else { > /* > * Keep removing sockets from the head until there's room for > * us to insert on the tail. In pre-locking revisions, this > * was a simple if(), but as we could be racing with other > * threads and soabort() requires dropping locks, we must > * loop waiting for the condition to be true. > */ > while (head->so_incqlen > head->so_qlimit) { > struct socket *sp; > sp = TAILQ_FIRST(&head->so_incomp); > TAILQ_REMOVE(&head->so_incomp, sp, so_list); > head->so_incqlen--; > sp->so_qstate &= ~SQ_INCOMP; > sp->so_head = NULL; > ACCEPT_UNLOCK(); > soabort(sp); > ACCEPT_LOCK(); > } > TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list); > so->so_qstate |= SQ_INCOMP; > head->so_incqlen++; > > I think the simplest approach would be to first convert as many places as > possible to use falloc_noinstall() / finstall(). If you end up with all of > them converted then you can just rename falloc_noinstall to falloc() and > retire the old falloc(). > I would expect soabort to result in a timeout/reset as opposed to regular connection close. Comments around soabort suggest it should not be used as a replacement for close, but maybe this is largely because of what the other end will see. That will need to be investigated. That said, I definitely support using delayed fd allocation (current falloc_noinstall) where possible, but I'm not convinced it is safe for all consumers. -- Mateusz Guzik From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 01:00:52 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D300EE72; Wed, 13 Aug 2014 01:00:52 +0000 (UTC) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 452932066; Wed, 13 Aug 2014 01:00:52 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id f8so73937wiw.0 for ; Tue, 12 Aug 2014 18:00:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=APSIwi3AQXGXeivwOXOxKI01UsBuX07vUJZozdwLxjI=; b=ZJA/pAKuKxktIrV11bUuZFoWyGQUNxk826zt74hKHbIcHUN4lb1c8ARlcZGFDC4a+A HeGoOvSiqsMSW2svL3yrH6lTJMeUI/KUhMDv7IHTobIcqDpz1+wQqtt99sWpZSrnxUKw Hp9lbXPviQ3cIgwxD3YKV2k5OHQqKadX0gt79Y6hIjxeNyIPvXj8TKLaE5D5m6bALP34 L3jiUUNrL1BUV3dIDQv+TZhZmDxhO9FR8c86n2kzHUuRpmdum2LTIua6fGM0EDdvruHY kq8/tmMpStCXlkwSHDT8YSD7aPM/TkEtw6gB+8zqL2uAqt7vxy4ejQkUyIhQ8Cx8+vvW dDKg== X-Received: by 10.180.206.84 with SMTP id lm20mr35747330wic.9.1407891650529; Tue, 12 Aug 2014 18:00:50 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id lq15sm2512779wic.1.2014.08.12.18.00.49 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Tue, 12 Aug 2014 18:00:49 -0700 (PDT) Date: Wed, 13 Aug 2014 03:00:46 +0200 From: Mateusz Guzik To: Konstantin Belousov Subject: Re: Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked Message-ID: <20140813010046.GB17869@dft-labs.eu> References: <20140713035500.GC16884@dft-labs.eu> <20140713132521.GY93733@kib.kiev.ua> <20140713133421.GA93733@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20140713133421.GA93733@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 01:00:52 -0000 On Sun, Jul 13, 2014 at 04:34:21PM +0300, Konstantin Belousov wrote: > On Sun, Jul 13, 2014 at 04:25:21PM +0300, Konstantin Belousov wrote: > > On Sun, Jul 13, 2014 at 05:55:00AM +0200, Mateusz Guzik wrote: > > > Currently: > > > /* > > > * Avoid reads reordering and then a first access to the > > > * fdp->fd_ofiles table which could result in OOB operation. > > > */ > > > if (fd < 0 || fd >= atomic_load_acq_int(&fdp->fd_nfiles)) > > > return (EBADF); > > > > > > However, if we put fd_nfiles and fd_otable into one atomically replaced > > > structure the only need to: > > > 1. make sure the pointer is read once > > > 2. issue a data dependency barrier - this is a noop on all supported > > > architectures and we don't even have approprate macro, so doing nothing > > > seems fine > > > > > > The motivation is to boost performance to amortize for seqlock cost, in > > > case it hits the tree. > > > > > > This has no impact on races with capability lookup. > > > > > > In a microbenchmark of 16 threads reading from the same pipe fd > > > immediately returning EAGAIN the numbers are: > > > x vanilla-readpipe-run-sum > > > + noacq-readpipe-run-sum > > > [..] > > > N Min Max Median Avg Stddev > > > x 20 13133671 14900364 13893331 13827075 471500.82 > > > + 20 59479718 59527286 59496714 59499504 13752.968 > > > Difference at 95.0% confidence > > > 4.56724e+07 +/- 213483 > > > 330.312% +/- 1.54395% > > > > > > There are 3 steps: > > > 1. tidy up capsicum to accept fde: > > > http://people.freebsd.org/~mjg/patches/single-fdtable-read-capsicum.patch > > > 2. add __READ_ONCE: > > > http://people.freebsd.org/~mjg/patches/read-once.patch > > > 3. put stuff into one structure: > > > http://people.freebsd.org/~mjg/patches/filedescenttable.patch > > > > > > Comments? > > > > We use 4-space indent for the continuation lines. Look at the malloc(9) > > call in the patch 3. > > > > The filedescenttable is really long name. Could it be, for instance, > > fdescenttbl ? > > > > Other than that, I think that the patches 2 and 3 are fine. I did not > > looked at the patch 1. > > > As an afterthought, you do not need __READ_ONCE(), the __DEVOLATILE() alone > would do what you need as well. Turns out patch 2 was quite bad. Reading http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf (pdf page 77) reveals: A cast of a value to a qualified type has no effect; the qualification (volatile, say) can have no effect on the access since it has occurred prior to the cast. If it is necessary to access a non-volatile object using volatile semantics, the technique is to cast the address of the object to the appropriate pointer-to-qualified type, then dereference that pointer. So how about we just follow the recomandation and also get the type automagically like linux folks do (added to sys/param.h): #define READ_ONCE(var) (*(volatile __typeof(var) *)&(var)) http://people.freebsd.org/~mjg/patches/read-once.patch I incorporated suggested changes have overwritten old patches. http://people.freebsd.org/~mjg/patches/filedescenttable.patch I would like to commit these changes this week with 2 weeks mfc. -- Mateusz Guzik From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 01:31:16 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5F3109F6; Wed, 13 Aug 2014 01:31:16 +0000 (UTC) Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D66123DF; Wed, 13 Aug 2014 01:31:16 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id i7so7969466oag.2 for ; Tue, 12 Aug 2014 18:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hkiTgSqRYYBbfFHp85rFYuLxqetNG+xgkIUPAASuRRA=; b=zOWAY9SNtOuA0Cr+8sj/VC5Cq7bKbw1+sLRPwAuvR6pzU72TQyKxElw8uthOgbzfBL B9Xc5QhqzTY2XMaCvrgNm5j+yk+quxwnKwkxzxaE6NaWKRfouLdE1kEt8EM1jLx/jkeo mS5uqshVCMI7RaBM1Wqd9SKNk9693BWXNpSNJJ+l44+Ha5Vm5du/1DEtltR52L/elHM9 8G1sJFTLFIIIRLoQyENtNB/rjv96JE8eaB7BCLyur1vVVgDCWzWFm4eyoMstrXeRQygA PyFVVtozKVgRAxXWC7QZgnnW9bJ2623+pVJvkQqgVS9fbTGxJ5w1Mh3Zj7NEa51+tcUA Nbdg== MIME-Version: 1.0 X-Received: by 10.182.249.52 with SMTP id yr20mr1082394obc.10.1407893475224; Tue, 12 Aug 2014 18:31:15 -0700 (PDT) Received: by 10.182.98.111 with HTTP; Tue, 12 Aug 2014 18:31:15 -0700 (PDT) In-Reply-To: <20140812233617.GA17869@dft-labs.eu> References: <20140717235538.GA15714@dft-labs.eu> <20140718155959.GN93733@kib.kiev.ua> <20140718191928.GB7179@dft-labs.eu> <201408111124.52064.jhb@freebsd.org> <20140812233617.GA17869@dft-labs.eu> Date: Tue, 12 Aug 2014 21:31:15 -0400 Message-ID: Subject: Re: current fd allocation idiom From: Benjamin Kaduk To: Mateusz Guzik Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 01:31:16 -0000 On Tue, Aug 12, 2014 at 7:36 PM, Mateusz Guzik wrote: > I would expect soabort to result in a timeout/reset as opposed to regular > connection close. > > Comments around soabort suggest it should not be used as a replacement > for close, but maybe this is largely because of what the other end will > see. That will need to be investigated. > > I added some text regarding soabort to socket.9 in r266962 -- does that help clarify the situation? -Ben From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 01:56:33 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4F9FA3C0; Wed, 13 Aug 2014 01:56:33 +0000 (UTC) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B58EA265C; Wed, 13 Aug 2014 01:56:32 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id ho1so6702344wib.14 for ; Tue, 12 Aug 2014 18:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=2SVmS/SN5+eeOjPquTCv4jEaX1wk+hGI/7Hja8hjaxs=; b=wEmBEaVWTwj5P61E25BPzJRB66Lq8kMQlFhba5Dex1vVeD/HQuIdBmlSt9sDs/W0Tc +PNU38Plt+8iIBazq1vnpDF6Xl2HAhR/36Stie3ae9QkXlbODvshq/M1cbzJLWGGuyAw CAaBY/xJBFIsQZGUPIA84uUC2Ue62y4kfEOcvWifRzq44n6CDllCX7q4G8eYhK7jfXdJ 6Cu+Z6OUSi/tuFPK3LMQmpwKeMl5D3Ojbf/fS10SaMLSITMZV2CsFBFMJzKYtzdcSdpF jMSk/f/YzkCMVP9+7T2MUPXkpJTvP9nJ7WJbix89sO2R7n2vIs59tsMeGKxX37n1McPG 2qvA== X-Received: by 10.180.94.34 with SMTP id cz2mr35788486wib.74.1407894990973; Tue, 12 Aug 2014 18:56:30 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id j5sm993076wjf.35.2014.08.12.18.56.29 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Tue, 12 Aug 2014 18:56:30 -0700 (PDT) Date: Wed, 13 Aug 2014 03:56:27 +0200 From: Mateusz Guzik To: Benjamin Kaduk Subject: Re: current fd allocation idiom Message-ID: <20140813015627.GC17869@dft-labs.eu> References: <20140717235538.GA15714@dft-labs.eu> <20140718155959.GN93733@kib.kiev.ua> <20140718191928.GB7179@dft-labs.eu> <201408111124.52064.jhb@freebsd.org> <20140812233617.GA17869@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 01:56:33 -0000 On Tue, Aug 12, 2014 at 09:31:15PM -0400, Benjamin Kaduk wrote: > On Tue, Aug 12, 2014 at 7:36 PM, Mateusz Guzik wrote: > > > I would expect soabort to result in a timeout/reset as opposed to regular > > connection close. > > > > Comments around soabort suggest it should not be used as a replacement > > for close, but maybe this is largely because of what the other end will > > see. That will need to be investigated. > > > > > I added some text regarding soabort to socket.9 in r266962 -- does that > help clarify the situation? > Nope. :-) It is unclear if the only motivation here is making sure nobody else sees the socket when given thread calls soabort. This would be easily guaranteed in here: fd allocation failed, fp with given socket was never exposed to anyone. So, if you say soabort would work here just fine, I'm happy to use it and blame you for problems. :-) -- Mateusz Guzik From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 03:31:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2BE1D594 for ; Wed, 13 Aug 2014 03:31:30 +0000 (UTC) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id CD05A200F for ; Wed, 13 Aug 2014 03:31:29 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 0820042270A; Wed, 13 Aug 2014 13:31:20 +1000 (EST) Date: Wed, 13 Aug 2014 13:31:17 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Mateusz Guzik Subject: Re: Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked In-Reply-To: <20140813010046.GB17869@dft-labs.eu> Message-ID: <20140813124428.K927@besplex.bde.org> References: <20140713035500.GC16884@dft-labs.eu> <20140713132521.GY93733@kib.kiev.ua> <20140713133421.GA93733@kib.kiev.ua> <20140813010046.GB17869@dft-labs.eu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=BdjhjNd2 c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=X_7xy1b9czwA:10 a=w8wdEOC8JkcA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=6I5d2MoRAAAA:8 a=b4LDLZbEAAAA:8 a=QiKvtI3h4XNE-eVjFlsA:9 a=4p95OmbumOrYGyu0:21 a=0sQSfhnAEEuDml_S:21 a=CjuIK1q_8ugA:10 Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 03:31:30 -0000 On Wed, 13 Aug 2014, Mateusz Guzik wrote: > On Sun, Jul 13, 2014 at 04:34:21PM +0300, Konstantin Belousov wrote: >> On Sun, Jul 13, 2014 at 04:25:21PM +0300, Konstantin Belousov wrote: >>> On Sun, Jul 13, 2014 at 05:55:00AM +0200, Mateusz Guzik wrote: >>>> Currently: >>>> /* >>>> * Avoid reads reordering and then a first access to the >>>> * fdp->fd_ofiles table which could result in OOB operation. >>>> */ >>>> if (fd < 0 || fd >= atomic_load_acq_int(&fdp->fd_nfiles)) >>>> return (EBADF); >>>> >>>> However, if we put fd_nfiles and fd_otable into one atomically replaced >>>> structure the only need to: >>>> 1. make sure the pointer is read once >>>> 2. issue a data dependency barrier - this is a noop on all supported >>>> architectures and we don't even have approprate macro, so doing nothing >>>> seems fine >>>> >>>> The motivation is to boost performance to amortize for seqlock cost, in >>>> case it hits the tree. >>>> >>>> This has no impact on races with capability lookup. >>>> >>>> In a microbenchmark of 16 threads reading from the same pipe fd >>>> immediately returning EAGAIN the numbers are: >>>> x vanilla-readpipe-run-sum >>>> + noacq-readpipe-run-sum >>>> [..] >>>> N Min Max Median Avg Stddev >>>> x 20 13133671 14900364 13893331 13827075 471500.82 >>>> + 20 59479718 59527286 59496714 59499504 13752.968 >>>> Difference at 95.0% confidence >>>> 4.56724e+07 +/- 213483 >>>> 330.312% +/- 1.54395% >>>> >>>> There are 3 steps: >>>> 1. tidy up capsicum to accept fde: >>>> http://people.freebsd.org/~mjg/patches/single-fdtable-read-capsicum.patch >>>> 2. add __READ_ONCE: >>>> http://people.freebsd.org/~mjg/patches/read-once.patch >>>> 3. put stuff into one structure: >>>> http://people.freebsd.org/~mjg/patches/filedescenttable.patch >>>> >>>> Comments? >>> >>> We use 4-space indent for the continuation lines. Look at the malloc(9) >>> call in the patch 3. >>> >>> The filedescenttable is really long name. Could it be, for instance, >>> fdescenttbl ? >>> >>> Other than that, I think that the patches 2 and 3 are fine. I did not >>> looked at the patch 1. >> >> As an afterthought, you do not need __READ_ONCE(), the __DEVOLATILE() alone >> would do what you need as well. > > Turns out patch 2 was quite bad. You pressed one of my hot keys (flame thrower). Anything that uses __DEVOLATILE() is bad. > Reading http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf > (pdf page 77) reveals: > A cast of a value to a qualified type has no effect; the qualification > (volatile, say) can have no effect on the access since it has occurred > prior to the cast. If it is necessary to access a non-volatile object > using volatile semantics, the technique is to cast the address of the > object to the appropriate pointer-to-qualified type, then dereference > that pointer. I wonder why it spells that out. It can go without saying that qualifiers only effect lvalues. > So how about we just follow the recomandation and also get the type > automagically like linux folks do (added to sys/param.h): > #define READ_ONCE(var) (*(volatile __typeof(var) *)&(var)) Casting this wat is indeed better. You have a variables that are usually locked by mutexes or another method. They are non-volatile then. But for some other accesses, they become volatile. READ_ONCE() is not a good name. I would have guessed it meant an atomic read. I think the ONCE in it just means "at least" once (instead of possibly zero times). The semantics of volatile are unclear except that they force the compiler to generate code that acts as if it forces the read. The comment in the patch says that it is to read the variable "only" once. Volatile semantics probably imply that too. But most uses don't require that. Reading volatile hardware status registers requires that, but for software you just want a non-cached value. > http://people.freebsd.org/~mjg/patches/read-once.patch Macros like this are broken for general use, since if the variable type already has a qualifier in it then adding the same qualifier gives too many qualifiers -- a constraint error. __DEVOLATILE() has similar bugs. It doesn't work on const-qualified types since it casts away the const in the first cast that it applies. That is just the first bug in it. I didn't fix this since I burned __DEVOLATILE() before it reached my tree. It is easier to fix that the above since it doesn't dereference anything. __typeof() is not very well designed since it doesn't allow breaking up a type into its components. Style bugs in the patch: - space instead of tab after #define. The previous macro visible in the patch has the same bug. The file has many other instances of this bug. That macro also spells __typeof() in gnu style (__typeof__()), and is missing parentheses for 'offset'. - sentence break of 1 space in the comment. Sentence breaks are 2 spaces in KNF. The file has only 2 other instances of this bug. - grammar and punctuation errors in "Useful e.g.". The sentence is missing an object, and I think commas are needd on each side of "e.g.", but that would be too formal so it would be better to simplify the wording a bit so as not to need so many commas. More bugs in the patch: - further undocumented namespace pollution. The previous macro visible in the patch has 2 underscores in its name so it doesn't have that bug. Both of thes macros may belong in where the underscores are more important. didn't have any with the underscores before the previous macro was added. Bruce From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 06:34:48 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 905D7117; Wed, 13 Aug 2014 06:34:48 +0000 (UTC) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4564025A8; Wed, 13 Aug 2014 06:34:48 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id x13so3745736qcv.40 for ; Tue, 12 Aug 2014 23:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=+iN+qOwkn4m+B2QoAd3nAlybwrDdaC7ShPmMRVeuGMk=; b=kttAUF4+B08aFYxRWPYp3FzsDF7YWyaBu0lyYFTR0RJG192MyEyjzenIax9TwSLTU4 FggL3Wi4/P6nSCKcEqEToBNCeMoSrxCShIRg+l9rA9ttnPX0Urq9tfgKvYqKuDJgZwtx ynZq2FHB+V1Lpp8ICzTVbzKO3427WRMnZfY6E3X7PljUO1dtT7oz11982DVt13TQYKSv Kz5l0wsR3IGbqJttIHsZF8inhrxQuLN/omgiewkyBQWGLmxCSQzZlgJnWpnbtuodN93Z ZLcdmN3/E8ZHbbjbKldbxiTEtqXDMOU/owA5N4Mj4HX+yLvfMgocOwtMEr8B+xVEDzU4 jrEA== MIME-Version: 1.0 X-Received: by 10.140.104.69 with SMTP id z63mr3227946qge.81.1407911687199; Tue, 12 Aug 2014 23:34:47 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.39.139 with HTTP; Tue, 12 Aug 2014 23:34:47 -0700 (PDT) Date: Tue, 12 Aug 2014 23:34:47 -0700 X-Google-Sender-Auth: qObJdXBICpTTmCtctbdnw9QYAj0 Message-ID: Subject: [rfc] make libbsdstat consumers work again From: Adrian Chadd To: Baptiste Daroussin , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 06:34:48 -0000 Hi! I hacked up something to make the libbsdstat consumers work. http://people.freebsd.org/~adrian/ath/20140812-bsdstat-1.diff How's that look? -a From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 16:17:28 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17C841DB; Wed, 13 Aug 2014 16:17:28 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EBA252B86; Wed, 13 Aug 2014 16:17:27 +0000 (UTC) Received: from [192.168.200.205] (c-50-131-5-126.hsd1.ca.comcast.net [50.131.5.126]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id C0EE4193892; Wed, 13 Aug 2014 16:17:20 +0000 (UTC) Subject: Re: [rfc] make libbsdstat consumers work again From: Sean Bruno Reply-To: sbruno@freebsd.org To: Adrian Chadd In-Reply-To: References: Content-Type: text/plain; charset="us-ascii" Date: Wed, 13 Aug 2014 09:17:18 -0700 Message-ID: <1407946638.1121.27.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Baptiste Daroussin , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 16:17:28 -0000 On Tue, 2014-08-12 at 23:34 -0700, Adrian Chadd wrote: > Hi! > > I hacked up something to make the libbsdstat consumers work. > > http://people.freebsd.org/~adrian/ath/20140812-bsdstat-1.diff > > How's that look? > > > -a Looks correct to me. Fixes wifi build errors that I've reported for mips. And builds here. sean From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 19:37:10 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1E314EFD; Wed, 13 Aug 2014 19:37:10 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0236.outbound.protection.outlook.com [207.46.163.236]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6298924E2; Wed, 13 Aug 2014 19:37:08 +0000 (UTC) Received: from BLUPR05CA0063.namprd05.prod.outlook.com (10.141.20.33) by BY2PR05MB726.namprd05.prod.outlook.com (10.141.223.19) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Wed, 13 Aug 2014 19:37:06 +0000 Received: from BY2FFO11FD058.protection.gbl (2a01:111:f400:7c0c::123) by BLUPR05CA0063.outlook.office365.com (2a01:111:e400:855::33) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Wed, 13 Aug 2014 19:37:04 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BY2FFO11FD058.mail.protection.outlook.com (10.1.15.178) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Wed, 13 Aug 2014 19:37:04 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 13 Aug 2014 12:36:30 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7DJaJn75280; Wed, 13 Aug 2014 12:36:22 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7DJaA1r089174; Wed, 13 Aug 2014 15:36:11 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408131936.s7DJaA1r089174@idle.juniper.net> To: Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML MIME-Version: 1.0 Content-ID: <89170.1407958568.0@idle.juniper.net> Date: Wed, 13 Aug 2014 15:36:10 -0400 From: Phil Shafer X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199003)(43234003)(189002)(164054003)(19580395003)(77982001)(4396001)(80022001)(74502001)(84676001)(83072002)(99396002)(71186001)(110136001)(79102001)(76482001)(54356999)(20776003)(84326002)(97736001)(19580405001)(50986999)(83322001)(69596002)(105596002)(86362001)(81156004)(68736004)(46102001)(512954002)(53416004)(87936001)(81542001)(81342001)(92726001)(85306004)(107046002)(21056001)(44976005)(85852003)(103666002)(6806004)(76506005)(74662001)(106466001)(95666004); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB726; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 0302D4F392 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net X-Mailman-Approved-At: Wed, 13 Aug 2014 20:40:01 +0000 Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 19:37:10 -0000 Phil Shafer writes: >FWIW, the UTF-8 strategy for libox is this: >- all format strings are UTF-8 >- argument strings (%s) are UTF-8 >- "%ls" handles wide characters >- "%hs" will handle locale-based strings >- XML, JSON, and HTML will be UTF-8 output >- text will be locale-based Sorry for the delay, but this code is now done. Formatting widths are done using wcwidth() so things like "%15.15s" work correctly regardless of locale settings. As a background task, I'm converting some basic commands to use libxo. It's slow work, but needs done.... I've a related topic: when an app goes to run a child command, how can it determine whether that binary supports libxo-based encoding requests? This should be known before the binary is run, since there's no means of auto-detecting the supported output after the fact. For example, say I want to make a JSON-based API for my server. I can setenv("LIBXO_OPTIONS", "json") to get JSON output, but I won't know if the binary supports this or if the output needs to be wrapped and escaped. I know ELF "Note" elements can be used to carry vendor-specific data, but have no experience with them. Would it be reasonable to use them as a means of communicating this information to other bits of software? Is FreeBSD using Notes for other information currently? Thanks, Phil P.s.: Attached is a screenshot of a quick demo using netstat output rendered in HTML with the jquery qtip popup that shows the XPath, along with some firebug output to show the contents. (The "data-qtip" attribute is added dynamically by the qtip library.) From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 21:13:24 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0D66EFB; Wed, 13 Aug 2014 21:13:24 +0000 (UTC) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id D27DF207C; Wed, 13 Aug 2014 21:13:24 +0000 (UTC) Received: from marvin.lab.vangyzen.net (c-24-125-214-90.hsd1.va.comcast.net [24.125.214.90]) by smtp.vangyzen.net (Postfix) with ESMTPSA id B4F8D56441; Wed, 13 Aug 2014 16:13:23 -0500 (CDT) Message-ID: <53EBD4F2.5080908@vangyzen.net> Date: Wed, 13 Aug 2014 17:13:22 -0400 From: Eric van Gyzen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Phil Shafer , Poul-Henning Kamp Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408131936.s7DJaA1r089174@idle.juniper.net> In-Reply-To: <201408131936.s7DJaA1r089174@idle.juniper.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 21:13:25 -0000 On 08/13/2014 15:36, Phil Shafer wrote: > Phil Shafer writes: >> FWIW, the UTF-8 strategy for libox is this: >> - all format strings are UTF-8 >> - argument strings (%s) are UTF-8 >> - "%ls" handles wide characters >> - "%hs" will handle locale-based strings >> - XML, JSON, and HTML will be UTF-8 output >> - text will be locale-based > Sorry for the delay, but this code is now done. Formatting widths > are done using wcwidth() so things like "%15.15s" work correctly > regardless of locale settings. As a background task, I'm converting > some basic commands to use libxo. It's slow work, but needs done.... > > I've a related topic: when an app goes to run a child command, how > can it determine whether that binary supports libxo-based encoding > requests? This should be known before the binary is run, since > there's no means of auto-detecting the supported output after the > fact. > > For example, say I want to make a JSON-based API for my server. I > can setenv("LIBXO_OPTIONS", "json") to get JSON output, but I won't > know if the binary supports this or if the output needs to be wrapped > and escaped. Perhaps libxo can have an API to answer this question, so your app can simply ask libxo if "netstat" supports libxo output. How should the API be implemented? Perhaps libxo can consult a file that lists all executables that support libxo. This file can be maintained either manually by committers or perhaps automatically by the build. (Maybe it could even be built into libxo.so itself, for efficiency, using a fallback to a file, for flexibility.) An alternative is to ask ELF if the executable is linked with libxo.so, but this obviously doesn't work when it's statically linked. I haven't given these much thought...just tossing out ideas. Eric > I know ELF "Note" elements can be used to carry vendor-specific > data, but have no experience with them. Would it be reasonable to > use them as a means of communicating this information to other bits > of software? Is FreeBSD using Notes for other information currently? From owner-freebsd-arch@FreeBSD.ORG Wed Aug 13 23:02:39 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EC9B5BF; Wed, 13 Aug 2014 23:02:39 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D5CD2C93; Wed, 13 Aug 2014 23:02:38 +0000 (UTC) Received: from rcannon-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s7DN2L02000748 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 13 Aug 2014 16:02:23 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_8B5850B6-5F0C-4912-9740-380033E110CC"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <201408131936.s7DJaA1r089174@idle.juniper.net> Date: Wed, 13 Aug 2014 16:02:15 -0700 Message-Id: <613EB1A5-2932-446A-A9A2-8CBDD060A00B@xcllnt.net> References: <201408131936.s7DJaA1r089174@idle.juniper.net> To: Phil Shafer X-Mailer: Apple Mail (2.1878.6) Cc: arch@freebsd.org, Poul-Henning Kamp , Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 23:02:39 -0000 --Apple-Mail=_8B5850B6-5F0C-4912-9740-380033E110CC Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Aug 13, 2014, at 12:36 PM, Phil Shafer wrote: > > I've a related topic: when an app goes to run a child command, how > can it determine whether that binary supports libxo-based encoding > requests? This should be known before the binary is run, since > there's no means of auto-detecting the supported output after the > fact. > > For example, say I want to make a JSON-based API for my server. I > can setenv("LIBXO_OPTIONS", "json") to get JSON output, but I won't > know if the binary supports this or if the output needs to be wrapped > and escaped. Aside: Using environment variables can be handy, but isn't always. What do you think about calling a libxo init function from main() and giving it argc and argv so that libxo options are parsed and removed just like what xlib does? > I know ELF "Note" elements can be used to carry vendor-specific > data, but have no experience with them. Would it be reasonable to > use them as a means of communicating this information to other bits > of software? Is FreeBSD using Notes for other information currently? Notes are used to tag the binary as a FreeBSD one (note is consumed by the kernel) or in core files for meta data (consumed by the debugger). A note section is definitely possible and reasonable. Especially if it's a note section for listing features. A special utility that consumes the note section to list features and returns whether a feature is supported is then very reasonable because it's generic. libxo would be the first feature we can check for. The question is: do we have more features we want to check for this way? If not, then such a scheme could be perceived as "heavy handed". Alternatives include looking for a particular symbol or possibly even running the utility with a libxo option that has predictable output. The last suggestion has some issues with handling the behaviour when libxo isn't supported therefore, a passive way to check seems better than having to run the utility. BTW: this is pretty powerful stuff! I feel FreeBSD is maturing :-) -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_8B5850B6-5F0C-4912-9740-380033E110CC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPr7ncACgkQpgWlLWHuifYjDwCdG0O211KFAV3e0kh25bp5r4Zj b3IAn0Y4qsFYAagnC3iiCm9DLUr0K9Y2 =A7HI -----END PGP SIGNATURE----- --Apple-Mail=_8B5850B6-5F0C-4912-9740-380033E110CC-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 04:52:44 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FAE01DB; Thu, 14 Aug 2014 04:52:44 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0206.outbound.protection.outlook.com [207.46.163.206]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E3FF2FD9; Thu, 14 Aug 2014 04:52:42 +0000 (UTC) Received: from CO2PR05MB729.namprd05.prod.outlook.com (10.141.228.12) by CO2PR05MB587.namprd05.prod.outlook.com (10.141.197.146) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Thu, 14 Aug 2014 04:52:26 +0000 Received: from BY2PR05CA003.namprd05.prod.outlook.com (10.242.32.33) by CO2PR05MB729.namprd05.prod.outlook.com (10.141.228.12) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Thu, 14 Aug 2014 04:52:25 +0000 Received: from BN1AFFO11FD052.protection.gbl (2a01:111:f400:7c10::115) by BY2PR05CA003.outlook.office365.com (2a01:111:e400:2c2a::33) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Thu, 14 Aug 2014 04:52:24 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1AFFO11FD052.mail.protection.outlook.com (10.58.53.67) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Thu, 14 Aug 2014 04:52:24 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 13 Aug 2014 21:52:23 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7E4qKn47003; Wed, 13 Aug 2014 21:52:20 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7E4qB2X091511; Thu, 14 Aug 2014 00:52:11 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408140452.s7E4qB2X091511@idle.juniper.net> To: Marcel Moolenaar Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <613EB1A5-2932-446A-A9A2-8CBDD060A00B@xcllnt.net> Date: Thu, 14 Aug 2014 00:52:11 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(164054003)(199003)(86362001)(44976005)(97736001)(103666002)(81342001)(19580395003)(50986999)(92726001)(6806004)(83322001)(74662001)(54356999)(102836001)(74502001)(68736004)(31966008)(69596002)(20776003)(46102001)(87936001)(81542001)(77982001)(85306004)(4396001)(110136001)(76506005)(21056001)(99396002)(107046002)(105596002)(95666004)(64706001)(48376002)(81156004)(50466002)(85852003)(92566001)(76482001)(79102001)(84676001)(83072002)(53416004)(106466001)(47776003)(80022001); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB729; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:;UriScan:; X-Forefront-PRVS: 03030B9493 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Poul-Henning Kamp , Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 04:52:44 -0000 Marcel Moolenaar writes: >Using environment variables can be handy, but isn't always. Yes, and there's a flag to turn this behaviour off (XOF_NO_ENV). An app that doesn't want this can turn it off at the top of main() using: xo_set_flags(NULL, XOF_NO_ENV); Also, the variable only affects the default handle, not any created by hand. >What do you think about calling a libxo init function from >main() and giving it argc and argv so that libxo options >are parsed and removed just like what xlib does? I don't want to presume that libxo's set of options are suitably distinct to allow safe canibalization of argv. But I do have a function (xo_set_options) that allows the app to pass options in opaquely, like: switch (getopt(..."X:"...)) { ... case 'X': if (xo_set_options(NULL, optarg) < 0) xo_errx(1, "invalid xo option: %s", optarg); break; ... But there's a chicken/egg problem, since these options need to be set before any text or errors are generated to ensure they are rendered in the right style. Something like: netstat --bad-option -X json,pretty wouldn't be making pretty json. Even then there could be issues like ld.so loading errors where having it set before loading begins makes sense (assuming ld.so gets munged to use libxo). >A note section is definitely possible and reasonable. >Especially if it's a note section for listing features. A >special utility that consumes the note section to list >features and returns whether a feature is supported is then >very reasonable because it's generic. libxo would be the >first feature we can check for. Cool. I'll give it a try. >The question is: do we have >more features we want to check for this way? If not, then >such a scheme could be perceived as "heavy handed". "heavy handed" in what sense? I'm hoping the note can be added by normal linker magic (but see question below). If not a "noteelf" command would need to be created (or an option to brandelf?) to mark the binary. Are you seeing something more? "elfdump" has a "-n" to dump notes, and a "-N " could be added, making "elfdump -n -N libxo my-app" the means of getting the contents. A "-q" option could be added to prevent output but set the exit code based on if the section appears in the given binary. >Alternatives include looking for a particular symbol or >possibly even running the utility with a libxo option that >has predictable output. How does one put a symbol in a binary when linking against a shared library? Would there need to be two libs, one with the code and one with just a symbol? I'd have the same issue with the ElfNote scheme, right? I'd need to add a section to the binary, but libxo could be linked dynamically. Is there an easy answer for this? Or is the app stuck with "LDADD+=-lxo -lxo-note"? >The last suggestion has some issues >with handling the behaviour when libxo isn't supported >therefore, a passive way to check seems better than having >to run the utility. I'd prefer a scheme where I can know before I run it what's needed. If the notes scheme is used generically, I could conceivably look further down the $PATH for a binary that supports my needs. Hmmm.... that would be a way to address the need to add an arch-based component to my $PATH in a scenario when $HOME is nfs mounted on machines with different architectures. >BTW: this is pretty powerful stuff! I feel FreeBSD is >maturing :-) Thanks. I need all the encouragement I can get; netstat alone has ~500 printf calls, many of which have me grepping kernel sources to determing sane field names. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 05:26:59 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4E90E7CB; Thu, 14 Aug 2014 05:26:59 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E22E72237; Thu, 14 Aug 2014 05:26:58 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7E5QmCG052829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 14 Aug 2014 08:26:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7E5QmCG052829 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7E5QmM4052828; Thu, 14 Aug 2014 08:26:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 14 Aug 2014 08:26:48 +0300 From: Konstantin Belousov To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140814052648.GM2737@kib.kiev.ua> References: <201408131936.s7DJaA1r089174@idle.juniper.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iUV/lbBrmPtUT9dM" Content-Disposition: inline In-Reply-To: <201408131936.s7DJaA1r089174@idle.juniper.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 05:26:59 -0000 --iUV/lbBrmPtUT9dM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 13, 2014 at 03:36:10PM -0400, Phil Shafer wrote: > Phil Shafer writes: > >FWIW, the UTF-8 strategy for libox is this: > >- all format strings are UTF-8 > >- argument strings (%s) are UTF-8 > >- "%ls" handles wide characters > >- "%hs" will handle locale-based strings > >- XML, JSON, and HTML will be UTF-8 output > >- text will be locale-based >=20 > Sorry for the delay, but this code is now done. Formatting widths > are done using wcwidth() so things like "%15.15s" work correctly > regardless of locale settings. As a background task, I'm converting > some basic commands to use libxo. It's slow work, but needs done.... >=20 > I've a related topic: when an app goes to run a child command, how > can it determine whether that binary supports libxo-based encoding > requests? This should be known before the binary is run, since > there's no means of auto-detecting the supported output after the > fact. >=20 > For example, say I want to make a JSON-based API for my server. I > can setenv("LIBXO_OPTIONS", "json") to get JSON output, but I won't > know if the binary supports this or if the output needs to be wrapped > and escaped. >=20 > I know ELF "Note" elements can be used to carry vendor-specific > data, but have no experience with them. Would it be reasonable to > use them as a means of communicating this information to other bits > of software? No. > Is FreeBSD using Notes for other information currently? Yes, the notes are used to communicate the information required by the dynamic linker to correctly activate the image. The mechanism has nothing to do with application-specific features, and overloading it for that purpose is severe and pointless layering violation. Things should not be done just because they could be done. Using the static tagging for the dynamic application properties is wrong anyway. E.g., would you consider the mere fact that the binary is linked against your library, as the indication that your feature is supported ? If not, how does it differ from the presence of some additional note ? --iUV/lbBrmPtUT9dM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT7EiXAAoJEJDCuSvBvK1BOKoP/1zGvwB4g3dkL+/oLInckVPD NEeNWyXgExYOCMkA9yNMhqagxT7t42VbvyacElOQb/kGdZImbKsxuqbdr+pWEOng nYb4EGG+ORS/21jkcEE84WwQrZzwNsnavr2NUJPQd4NML19Y+zKGwIpLK+oguYKM sILBXLxVmmI2Z/ef8O7J8l69MsaP+llPo5lOgcFE6D/DsAjSNQyRSFAMDODB3V7+ na1rGqNSvQNfNR5w3YQUYtSo/htDKy4hkZphayQTP68EFE8gM5lv7pqIV3GABvq1 dIjDDhzTAtlsi2U5Z4cL/RV+bWHYwKYatHAgJKFiuMx+MkOw/GLXOTVO0kZcQyV4 OGVVEZ+xURnKNaHYaozy520EoYlCz4rkOGvP8O9yiB0qdhjXDF1RJptlGByx30K0 4fRA7vKG61Aks92XR+jezhu0AcLXcKnighgi46L+/HfEYM63Tqoyvm6MkE1JVMb4 idkKxO4dH/91CXv/CeNYpSeZLtfXtaT6Y2oknlh5+uJlVOYS2Dg/ambwcyDqowjb qGHGy6PQz9kdEAbTj16LYTWpPNlpNXHy7r6n7P4ufIB0lsnVFw6mrJ5XGmzoRe3D SmxSS0eg4oMsOargh/W5DxkyRg5Wk3cCFZU8j4g7kpvmN6sIqShP8Gcp6/qqUKEd lXhD6rit2OI265CDBbGY =2x00 -----END PGP SIGNATURE----- --iUV/lbBrmPtUT9dM-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 06:07:24 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3598E5CF; Thu, 14 Aug 2014 06:07:24 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0243.outbound.protection.outlook.com [207.46.163.243]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB9D82644; Thu, 14 Aug 2014 06:07:23 +0000 (UTC) Received: from CO2PR05CA014.namprd05.prod.outlook.com (10.141.241.142) by BY2PR05MB725.namprd05.prod.outlook.com (10.141.223.13) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Thu, 14 Aug 2014 06:07:13 +0000 Received: from BN1BFFO11FD036.protection.gbl (2a01:111:f400:7c10::1:148) by CO2PR05CA014.outlook.office365.com (2a01:111:e400:1429::14) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Thu, 14 Aug 2014 06:07:12 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1BFFO11FD036.mail.protection.outlook.com (10.58.144.99) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Thu, 14 Aug 2014 06:07:10 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 13 Aug 2014 23:06:47 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7E66gn92493; Wed, 13 Aug 2014 23:06:44 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7E66XXA091972; Thu, 14 Aug 2014 02:06:33 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408140606.s7E66XXA091972@idle.juniper.net> To: Konstantin Belousov Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140814052648.GM2737@kib.kiev.ua> Date: Thu, 14 Aug 2014 02:06:33 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(164054003)(199003)(74662001)(46102001)(50986999)(76506005)(99396002)(31966008)(21056001)(74502001)(79102001)(76482001)(53416004)(68736004)(69596002)(44976005)(81156004)(6806004)(54356999)(105596002)(77982001)(103666002)(86362001)(107046002)(83322001)(1411001)(102836001)(48376002)(20776003)(95666004)(50466002)(85306004)(92726001)(64706001)(47776003)(4396001)(92566001)(87936001)(106466001)(85852003)(83072002)(97736001)(84676001)(80022001)(81542001)(110136001)(81342001); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB725; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 03030B9493 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 06:07:24 -0000 Konstantin Belousov writes: >Yes, the notes are used to communicate the information required by >the dynamic linker to correctly activate the image. The mechanism has >nothing to do with application-specific features, and overloading it for >that purpose is severe and pointless layering violation. The ELF spec says: Note Section Sometimes a vendor or system builder needs to mark an object file with special information that other programs will check for conformance, compatibility, etc. Sections of type SHT_NOTE and program header elements of type PT_NOTE can be used for this purpose. The note information in sections and program header elements holds any number of entries, each of which is an array of 4-byte words in the format of the target processor. Labels appear below to help explain note information organization, but they are not part of the specification. Marking the binary with a libxo-specific note tells the caller that the binary is capable of rendering its output in a non-traditional style and gives the caller a means of triggering those styles of output. In the libxo-enabled world, I see this as vital information the caller needs to initialize the environment in which the command will be run. Isn't this exactly the sort of information ELF targets for note sections? Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 08:53:07 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9EC6292F; Thu, 14 Aug 2014 08:53:07 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 254EA271A; Thu, 14 Aug 2014 08:53:06 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7E8qwvc099612 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 14 Aug 2014 11:52:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7E8qwvc099612 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7E8qvqE099611; Thu, 14 Aug 2014 11:52:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 14 Aug 2014 11:52:57 +0300 From: Konstantin Belousov To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140814085257.GN2737@kib.kiev.ua> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aSnC4ZPPfhCvD8sN" Content-Disposition: inline In-Reply-To: <201408140606.s7E66XXA091972@idle.juniper.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 08:53:07 -0000 --aSnC4ZPPfhCvD8sN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 14, 2014 at 02:06:33AM -0400, Phil Shafer wrote: > Konstantin Belousov writes: > >Yes, the notes are used to communicate the information required by > >the dynamic linker to correctly activate the image. The mechanism has > >nothing to do with application-specific features, and overloading it for > >that purpose is severe and pointless layering violation. >=20 > The ELF spec says: >=20 > Note Section >=20 > Sometimes a vendor or system builder needs to mark an object > file with special information that other programs will check > for conformance, compatibility, etc. Sections of type SHT_NOTE > and program header elements of type PT_NOTE can be used for > this purpose. The note information in sections and program > header elements holds any number of entries, each of which is > an array of 4-byte words in the format of the target processor. > Labels appear below to help explain note information organization, > but they are not part of the specification. ELF standard scope is about build toolchain and C runtime, where the cited paragraph makes perfect sense. >=20 > Marking the binary with a libxo-specific note tells the caller that > the binary is capable of rendering its output in a non-traditional > style and gives the caller a means of triggering those styles of > output. In the libxo-enabled world, I see this as vital information > the caller needs to initialize the environment in which the command > will be run. Isn't this exactly the sort of information ELF targets > for note sections? How binary format has any relevance for an application level feature ? What would you do with the binaries which permissions are 'r-s--x--x', which is not unexpected for the tools which gather system information and have to access things like /dev/mem ? You removed and did not answered a crusial question, which is a litmus test for your proposal. Namely, how presence of the proposed note in the binary is different from DT_NEEDED tag for your library ? Definitely, I do not see an addition of the fashion-of-the-day text-mangling output shattering enough to justify imposing the architecture violation. --aSnC4ZPPfhCvD8sN Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT7HjpAAoJEJDCuSvBvK1BlY4P+gK4CHB71vO90ziLrLhVQm/m kTVLOoMKQoce46xHoeoePXHXBDzzurPKZUI9DyCjOTdNyjAxKvNm8yNb8TQrDSpd ASn4ztWJih5WHrOr1dAXwKLl6RKIDzUUQGvYQfpZbMgLQhLC7jHnC67v42YDN7CT 0DYY/VbSilvgThUNSfhvK3eQp846lOb2T6dEoc3/dDSeoM0WnkZnlpI//ztsy1wG U6hdolglbCm28rFvB3eMB2gIv6uGcwxjFn7EutkLZ4Qs7cxtG1dzFEpGIEDiBSfc Sr4hqLHcqnbPwAZdWGfhuKYBhOZaC4SAA4CefeBYNMhobYV+gFAFhuWlnkh6g5J+ riDupZcUM9e+YTbu2C2XxBGaTwIyCF7ESps89x5yhb46WikgVzjj9yNbh4VgAZk3 7EmsWaogZR6qRNKQdZimn28ksN42J/Ryu1PL7KG/e6cWK2b/kj2A2OmB78eBgEgy uKbzFD4XmCCkAEkbgl4D/LBZvEa1wRsnQaNSzMaChv0RCToixkaFdKAKqXbi3bvG RdRWZnaGqSxA21JnZWgcMDGuBNqpGt4yYwGxOebdCPcgi2TTik/PpqmYq4He2PyC 7ukjFTx6+RbFZlplIN7z/V2B4iKL/tlBWHbThBzC1FzNlsV1f8JQNlPTmFZado6C TYNJm4D056mDdGTTYieK =ANxL -----END PGP SIGNATURE----- --aSnC4ZPPfhCvD8sN-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 14:08:53 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1321AE8F; Thu, 14 Aug 2014 14:08:53 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C65432A9A; Thu, 14 Aug 2014 14:08:52 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1XHvhx-000KVa-6r; Thu, 14 Aug 2014 14:08:49 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s7EE8lhZ037675; Thu, 14 Aug 2014 08:08:47 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+/Mpyv0Pz48DNgNaJdfRV/ X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Ian Lepore To: Konstantin Belousov In-Reply-To: <20140814085257.GN2737@kib.kiev.ua> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> Content-Type: text/plain; charset="us-ascii" Date: Thu, 14 Aug 2014 08:08:46 -0600 Message-ID: <1408025326.56408.566.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: marcel@freebsd.org, Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 14:08:53 -0000 On Thu, 2014-08-14 at 11:52 +0300, Konstantin Belousov wrote: > On Thu, Aug 14, 2014 at 02:06:33AM -0400, Phil Shafer wrote: > > Konstantin Belousov writes: > > >Yes, the notes are used to communicate the information required by > > >the dynamic linker to correctly activate the image. The mechanism has > > >nothing to do with application-specific features, and overloading it for > > >that purpose is severe and pointless layering violation. > > > > The ELF spec says: > > > > Note Section > > > > Sometimes a vendor or system builder needs to mark an object > > file with special information that other programs will check > > for conformance, compatibility, etc. Sections of type SHT_NOTE > > and program header elements of type PT_NOTE can be used for > > this purpose. The note information in sections and program > > header elements holds any number of entries, each of which is > > an array of 4-byte words in the format of the target processor. > > Labels appear below to help explain note information organization, > > but they are not part of the specification. > ELF standard scope is about build toolchain and C runtime, where the > cited paragraph makes perfect sense. > I disagree with this interpretation. The cited paragraph can be found in, for example, the Oracle documentation in a chapter named "object file format". There is nothing about the context that limits the validity to toolchains and runtime support, it's just describing the file layout. It appears to me that the NOTE mechanism is purposely designed for attaching arbitrary metadata of any size and type to an elf file. A bit of searching doesn't turn up any words that either recommend or forbid certain types of info stored in NOTEs. > > > > Marking the binary with a libxo-specific note tells the caller that > > the binary is capable of rendering its output in a non-traditional > > style and gives the caller a means of triggering those styles of > > output. In the libxo-enabled world, I see this as vital information > > the caller needs to initialize the environment in which the command > > will be run. Isn't this exactly the sort of information ELF targets > > for note sections? > > How binary format has any relevance for an application level feature ? > What would you do with the binaries which permissions are 'r-s--x--x', > which is not unexpected for the tools which gather system information > and have to access things like /dev/mem ? > > You removed and did not answered a crusial question, which is a litmus > test for your proposal. Namely, how presence of the proposed note in > the binary is different from DT_NEEDED tag for your library ? DT_NEEDED only helps with dynamically linked executables, this whole NOTEs discussion came up in the context of how to detect a statically linked binary with libxo output support. > > Definitely, I do not see an addition of the fashion-of-the-day > text-mangling output shattering enough to justify imposing the > architecture violation. I don't think you've cited anything other than your own opinion that using a note is any sort of architecture violation. I don't know that you're wrong, I just can't find anything with a bit of quick searching that says you're right. -- Ian From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 15:17:08 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD37F967; Thu, 14 Aug 2014 15:17:08 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0190.outbound.protection.outlook.com [207.46.163.190]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B7DF2425; Thu, 14 Aug 2014 15:17:07 +0000 (UTC) Received: from BY2PR05CA009.namprd05.prod.outlook.com (10.242.32.39) by DM2PR05MB734.namprd05.prod.outlook.com (10.141.178.22) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 14 Aug 2014 15:16:59 +0000 Received: from BL2FFO11FD027.protection.gbl (2a01:111:f400:7c09::163) by BY2PR05CA009.outlook.office365.com (2a01:111:e400:2c2a::39) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Thu, 14 Aug 2014 15:16:58 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BL2FFO11FD027.mail.protection.outlook.com (10.173.161.106) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Thu, 14 Aug 2014 15:16:57 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 14 Aug 2014 08:16:36 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7EFGOn86702; Thu, 14 Aug 2014 08:16:29 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7EFGE4a096197; Thu, 14 Aug 2014 11:16:14 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408141516.s7EFGE4a096197@idle.juniper.net> To: Konstantin Belousov Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140814085257.GN2737@kib.kiev.ua> Date: Thu, 14 Aug 2014 11:16:14 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199003)(189002)(164054003)(561944003)(87936001)(81542001)(77982001)(74662001)(102836001)(110136001)(81342001)(107046002)(83072002)(74502001)(80022001)(48376002)(6806004)(53416004)(69596002)(106466001)(4396001)(86362001)(92726001)(81156004)(85306004)(84676001)(1411001)(76482001)(92566001)(50986999)(103666002)(64706001)(46102001)(54356999)(44976005)(105596002)(95666004)(83322001)(79102001)(21056001)(97736001)(76506005)(20776003)(85852003)(47776003)(99396002)(31966008)(50466002)(68736004); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB734; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 03030B9493 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, Poul-Henning Kamp , marcel@freebsd.org, John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 15:17:08 -0000 Konstantin Belousov writes: >How binary format has any relevance for an application level feature ? >What would you do with the binaries which permissions are 'r-s--x--x', >which is not unexpected for the tools which gather system information >and have to access things like /dev/mem ? This would clearly not make sense. Some meta-data should be in the file and some in the filesystem. Implementing the SF_SNAPSHOT file as a note section would be silly. But that doesn't imply that using a note section to facilitate proper construction of the environment for running a binary isn't reasonable. >You removed and did not answered a crusial question, which is a litmus >test for your proposal. Namely, how presence of the proposed note in >the binary is different from DT_NEEDED tag for your library ? Apologies; here is your original question: >>Using the static tagging for the dynamic application properties is wrong >>anyway. E.g., would you consider the mere fact that the binary is linked >>against your library, as the indication that your feature is supported ? >>If not, how does it differ from the presence of some additional note ? No, I'm not looking for something more explicit than a reference to a function in a library. I'm looking for an explicit marker that a binary supports working in a particular environment. That marker could be applied by having the developer link against a specific marking library, or by having a tool make the binary appropriately. But it should be something explicit. Re: DT_NEEDED: this section holds symbols for dynamic linking. It's content and meaning are explicitly given in the spec. The note section is intended for other generic information. It seems a reasonable place to put the answer to the question "can this binary make additional styles of output and how do I trigger that behavior?". >Definitely, I do not see an addition of the fashion-of-the-day >text-mangling output shattering enough to justify imposing the >architecture violation. It's partially opinion and perspective, but I don't see an architecture violation; I see the use of a generic mechanism to carry relevant information. And I see this addition as a modernization that allows better integration with fashionable tools like browsers and client/server architectures. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 15:24:11 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB85FDE8 for ; Thu, 14 Aug 2014 15:24:11 +0000 (UTC) Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55E242508 for ; Thu, 14 Aug 2014 15:24:11 +0000 (UTC) Received: by mail-ig0-f172.google.com with SMTP id h15so13860370igd.5 for ; Thu, 14 Aug 2014 08:24:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=4E2RqfxhYsX/xyenQjLzzFNm09Bh6LZGoYIJjqwz6xU=; b=AAggTa06rjTTKR37MKnYhmPM634i49YsDYeHymzNaqyd+FOvwCi/ubcoS2WTZA2rq3 AOHoBuoLRCDDy7UEoaws4HCQab78NIZ6I5dBDhEsRmV51AStz31ifBSdF93rZQrOjql2 lel5IFFspkkZjwrRxp6hBkcCWISUSUaCKPmTdRMn3u6yOvb6oR9rDgnpl4VB+oX/iUaj 6fOJGUUI8CKznX35XZACBv4m2CdMT1HXhAnVHHpOtfDsZDaI9Q0h56DW6N1cny7lXahS omvPsdmGzynpp7iNPqJPZK2mMoDj2szked2Iha6xax2wiibiDRzzdUzKSJcDWdLPSpTo ATuA== X-Gm-Message-State: ALoCoQnlTQQ7RYIE/DJxzG+8mAHWDJ0ATyKblj5rbfU1Qc02Sl4DjffXEKL9A0hmXuRcNNUZSkO1 X-Received: by 10.50.61.145 with SMTP id p17mr17770128igr.41.1408029842333; Thu, 14 Aug 2014 08:24:02 -0700 (PDT) Received: from bsdimp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id nn13sm21379206igb.7.2014.08.14.08.23.59 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 14 Aug 2014 08:24:00 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_086615EE-779B-42D1-901B-9B8227B36641"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Warner Losh In-Reply-To: <201408141516.s7EFGE4a096197@idle.juniper.net> Date: Thu, 14 Aug 2014 09:23:58 -0600 Message-Id: <746AC443-B255-47DD-8C24-3E3A32A5CC05@bsdimp.com> References: <201408141516.s7EFGE4a096197@idle.juniper.net> To: Phil Shafer X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 15:24:11 -0000 --Apple-Mail=_086615EE-779B-42D1-901B-9B8227B36641 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Sorry for top posting, this really isn=92t responsive to the minutia in = the rest of the thread. I=92m curious. Why isn=92t this conversation about =93foo =97supports-xml=94= ? Why tag these commands with weird, non-standard things that need more = exotic tools to dig the information out. Why not have a standardized = command line option that prints nothing and returns 0 for success, or = whines and returns 1 for failure? That=92s way more standardized than = adding obscure notes that may or may not be allowed by the standard, but = that we traditionally haven=92t done, which requires tools that aren=92t = standardized and whose interface varies from one tool to the next. This = is true of asking about DT_NEEDED (which forces a specific library for = the implementation) as well as anything placed in the NOTES section. It = also assumes that you know the thing you are querying is an ELF = executable, that you can find it, that there=92s not a shell script = wrapper for that tool that redirects to binaries that do support this, = etc, etc etc. Basically, what does this =91meta data=92 really buy you that can=92t be = bought some other, more standard, more direct way that doesn=92t = enshrine so many hard-coded implementation decisions into the mix? Warner On Aug 14, 2014, at 9:16 AM, Phil Shafer wrote: > Konstantin Belousov writes: >> How binary format has any relevance for an application level feature = ? >> What would you do with the binaries which permissions are = 'r-s--x--x', >> which is not unexpected for the tools which gather system information >> and have to access things like /dev/mem ? >=20 > This would clearly not make sense. Some meta-data should be > in the file and some in the filesystem. Implementing the > SF_SNAPSHOT file as a note section would be silly. But > that doesn't imply that using a note section to facilitate > proper construction of the environment for running a binary > isn't reasonable. >=20 >> You removed and did not answered a crusial question, which is a = litmus >> test for your proposal. Namely, how presence of the proposed note in >> the binary is different from DT_NEEDED tag for your library ? >=20 > Apologies; here is your original question: >=20 >>> Using the static tagging for the dynamic application properties is = wrong >>> anyway. E.g., would you consider the mere fact that the binary is = linked >>> against your library, as the indication that your feature is = supported ? >>> If not, how does it differ from the presence of some additional note = ? >=20 > No, I'm not looking for something more explicit than a reference > to a function in a library. I'm looking for an explicit marker > that a binary supports working in a particular environment. That > marker could be applied by having the developer link against a > specific marking library, or by having a tool make the binary > appropriately. But it should be something explicit. >=20 > Re: DT_NEEDED: this section holds symbols for dynamic linking. It's > content and meaning are explicitly given in the spec. The note > section is intended for other generic information. It seems a > reasonable place to put the answer to the question "can this binary > make additional styles of output and how do I trigger that behavior?". >=20 >> Definitely, I do not see an addition of the fashion-of-the-day >> text-mangling output shattering enough to justify imposing the >> architecture violation. >=20 > It's partially opinion and perspective, but I don't see an = architecture > violation; I see the use of a generic mechanism to carry relevant > information. And I see this addition as a modernization that allows > better integration with fashionable tools like browsers and > client/server architectures. >=20 > Thanks, > Phil > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to = "freebsd-arch-unsubscribe@freebsd.org" --Apple-Mail=_086615EE-779B-42D1-901B-9B8227B36641 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJT7NSOAAoJEGwc0Sh9sBEAr9IQAMgYGxnStDoK9WpJGDuXE2Y7 bbLn4Abot42L5xlw7LDOJkT5EJA30uU84fqeA9qECLn2pPEV+9jyWJATUekRNlew Z8frd2WaB7od5soNdfGgmhEQFjYWNZb+fu6Kuk3X1E6cmVaEDGyb0SRh8mE6PW58 0gwSn1N3mAH8dvM0T3TclGezv9kR40zwOpVxXpVgyOCgfp4xSd0ySXE5MMZosVOQ N9Rnpf4XbzPFL04mNUN0kpgpUzRZtS3br28uh6kK0hylZ1ShebSpCN9QlCBOEwFM nix+Dsd1Y1XfDGGaiFWHpxM/uqplQaqHJnSzml0BUPSAl7ifckF8NBVhiED/NF1M iOh792EmRxSYRRwOtk3CSsUdrUPTSzCcuNxd40S31aJVt10Y3G47pTJPeeI7grbd Ad3eZAJe1N/QnIzxWMk6v7S+bZ99qy3ZjQ/KWgHZL88+u0/1lOmkW7PxdVm7TPiU 7njYKF8K0DGkkR0hyNO335c+0FOkms1N8ZtLFax8sMEGxgqeV9ssBQS9+vsxsj9j VkuCip2TQ6pfR8NWbkBwfMQFY4SWHAZzUcK0hoTprdkk4jZY9OoIRcSUIKFWhM7w xtHl56qp6SApjfnPUjq3dQp/tZllHSap4Q4rLzvXcgn/uaszo0m/BuLU/uUOpVSV f4ubkQ4haRuS3hzYZW47 =5OGR -----END PGP SIGNATURE----- --Apple-Mail=_086615EE-779B-42D1-901B-9B8227B36641-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:07:13 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0EA6124B; Thu, 14 Aug 2014 16:07:13 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA7BB29A3; Thu, 14 Aug 2014 16:07:12 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 1F601B917; Thu, 14 Aug 2014 12:07:11 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked Date: Thu, 14 Aug 2014 08:41:47 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140713035500.GC16884@dft-labs.eu> In-Reply-To: <20140713035500.GC16884@dft-labs.eu> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408140841.47510.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:11 -0400 (EDT) Cc: Mateusz Guzik , kib@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:13 -0000 On Saturday, July 12, 2014 11:55:00 pm Mateusz Guzik wrote: > Currently: > /* > * Avoid reads reordering and then a first access to the > * fdp->fd_ofiles table which could result in OOB operation. > */ > if (fd < 0 || fd >= atomic_load_acq_int(&fdp->fd_nfiles)) > return (EBADF); > > However, if we put fd_nfiles and fd_otable into one atomically replaced > structure the only need to: > 1. make sure the pointer is read once > 2. issue a data dependency barrier - this is a noop on all supported > architectures and we don't even have approprate macro, so doing nothing > seems fine > > The motivation is to boost performance to amortize for seqlock cost, in > case it hits the tree. > > This has no impact on races with capability lookup. > > In a microbenchmark of 16 threads reading from the same pipe fd > immediately returning EAGAIN the numbers are: > x vanilla-readpipe-run-sum > + noacq-readpipe-run-sum > [..] > N Min Max Median Avg Stddev > x 20 13133671 14900364 13893331 13827075 471500.82 > + 20 59479718 59527286 59496714 59499504 13752.968 > Difference at 95.0% confidence > 4.56724e+07 +/- 213483 > 330.312% +/- 1.54395% > > There are 3 steps: > 1. tidy up capsicum to accept fde: > http://people.freebsd.org/~mjg/patches/single-fdtable-read-capsicum.patch The KASSERT() on the 'fd' being valid was lost from cap_fcntl_check(). Should probably put that back. However, cap_fcntl_check() is now no longer used. Instead, you should just rename cap_fcntl_check_fd() to cap_fcntl_check() and remove the old one. > 2. add __READ_ONCE: > http://people.freebsd.org/~mjg/patches/read-once.patch > 3. put stuff into one structure: > http://people.freebsd.org/~mjg/patches/filedescenttable.patch Anotehr name besides the one suggested by Konstantin could just be 'struct filetable'. Even shorter while not having abbreviations. Also, I think you can leave the comment in 'struct filedesc' as just 'open files' in that case. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:07:14 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A1DF24D; Thu, 14 Aug 2014 16:07:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EEE729A4; Thu, 14 Aug 2014 16:07:14 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id F225AB922; Thu, 14 Aug 2014 12:07:12 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Thu, 14 Aug 2014 08:47:00 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> In-Reply-To: <20140814085257.GN2737@kib.kiev.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201408140847.00573.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:13 -0400 (EDT) Cc: marcel@freebsd.org, Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:14 -0000 On Thursday, August 14, 2014 4:52:57 am Konstantin Belousov wrote: > On Thu, Aug 14, 2014 at 02:06:33AM -0400, Phil Shafer wrote: > > Konstantin Belousov writes: > > >Yes, the notes are used to communicate the information required by > > >the dynamic linker to correctly activate the image. The mechanism has > > >nothing to do with application-specific features, and overloading it for > > >that purpose is severe and pointless layering violation. > > > > The ELF spec says: > > > > Note Section > > > > Sometimes a vendor or system builder needs to mark an object > > file with special information that other programs will check > > for conformance, compatibility, etc. Sections of type SHT_NOTE > > and program header elements of type PT_NOTE can be used for > > this purpose. The note information in sections and program > > header elements holds any number of entries, each of which is > > an array of 4-byte words in the format of the target processor. > > Labels appear below to help explain note information organization, > > but they are not part of the specification. > ELF standard scope is about build toolchain and C runtime, where the > cited paragraph makes perfect sense. Agreed. > > Marking the binary with a libxo-specific note tells the caller that > > the binary is capable of rendering its output in a non-traditional > > style and gives the caller a means of triggering those styles of > > output. In the libxo-enabled world, I see this as vital information > > the caller needs to initialize the environment in which the command > > will be run. Isn't this exactly the sort of information ELF targets > > for note sections? > > How binary format has any relevance for an application level feature ? > What would you do with the binaries which permissions are 'r-s--x--x', > which is not unexpected for the tools which gather system information > and have to access things like /dev/mem ? > > You removed and did not answered a crusial question, which is a litmus > test for your proposal. Namely, how presence of the proposed note in > the binary is different from DT_NEEDED tag for your library ? Yes, checking DT_NEEDED for libxo.so is the first thing I thought of as well. It is equivalent to 'ldd foo | grep libxo'. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:07:19 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83E1940D for ; Thu, 14 Aug 2014 16:07:19 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A73229A9 for ; Thu, 14 Aug 2014 16:07:19 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4E098B9BA; Thu, 14 Aug 2014 12:07:18 -0400 (EDT) From: John Baldwin To: Adrian Chadd Subject: Re: RFC: cpuid_t Date: Thu, 14 Aug 2014 11:43:01 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <201408120939.56176.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201408141143.01137.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:18 -0400 (EDT) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:19 -0000 On Tuesday, August 12, 2014 6:17:27 pm Adrian Chadd wrote: > On 12 August 2014 06:39, John Baldwin wrote: > > > I think Bruce's point is you can just fix the few places that use a u_char to > > an int and call it a day. The biggest offender of those that I can think of > > are td_oncpu/td_lastcpu and the corresponding fields in kinfo along with > > expanding NOCPU to be -1 instead of 255. I think most other places already > > use an int (pf is one place that doesn't, it borrows N bits from some other > > field to store the CPU number, so it can't use a cpuid_t anyway), so the patch > > to just use an int might actually be quite small. > > You could do that, but then we'd be left in a situation that it isn't > 100% clear what type people should be using (except hoping they > cut/paste code that uses an int and don't make dumb decisions like > assuming they can pack it into other fields, like pf) and that they > could treat the CPU ID as some kind of index number. Being able to > grep for all the places where cpuid_t is will make it a lot easier in > the future to uplift all of the cpu id manipulating code to treat CPU > IDs differently (eg making things very sparse.) I'm not sure how a typedef makes it easier to handle sparse IDs over an int? Plus, the IDs are already allowed to be sparse (hence CPU_ABSENT() and CPU_FOREACH(), etc.). I doubt that a typedef would make it any easier to deal with issues like pf either because the current code is not explicitly using any type at all, it's just passed into magic macros. > So, I'm happy if the group decision is to just make it an int. It just > has to be done before FreeBSD-11 ships. :) I think making it an int would be a good first step at the very least. It would be good to see how many places don't already use it that way. > I don't have such an aversion to typedefs. It has always made it much > easier to do type checking and changes to things in C that I'd > normally do in C++ with classes. One downside is that for printf() (a non-trivial thing in C), you have to either violate the abstraction (i.e. "know" it's an int) or use intmax_t casts everywhere. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:07:20 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53409414 for ; Thu, 14 Aug 2014 16:07:20 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 258F429AA for ; Thu, 14 Aug 2014 16:07:20 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 05913B9C3; Thu, 14 Aug 2014 12:07:19 -0400 (EDT) From: John Baldwin To: Mateusz Guzik Subject: Re: current fd allocation idiom Date: Thu, 14 Aug 2014 11:44:50 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140717235538.GA15714@dft-labs.eu> <201408111124.52064.jhb@freebsd.org> <20140812233617.GA17869@dft-labs.eu> In-Reply-To: <20140812233617.GA17869@dft-labs.eu> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201408141144.51131.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:19 -0400 (EDT) Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:20 -0000 On Tuesday, August 12, 2014 7:36:18 pm Mateusz Guzik wrote: > On Mon, Aug 11, 2014 at 11:24:51AM -0400, John Baldwin wrote: > > On Friday, July 18, 2014 3:19:28 pm Mateusz Guzik wrote: > > > On Fri, Jul 18, 2014 at 06:59:59PM +0300, Konstantin Belousov wrote: > > > > On Fri, Jul 18, 2014 at 04:40:12PM +0200, Mateusz Guzik wrote: > > > > > On Fri, Jul 18, 2014 at 04:06:29PM +0300, Konstantin Belousov wrote: > > > > > > It seems that all what is needed is conversion of places using > > > > > > falloc() to falloc_noinstall()/finstall(). > > > > > > > > > > > > > > > > This postpones fd allocation to after interested function did all work > > > > > it wanted to do, which means we would need reliable ways of reverting > > > > > all the work in case allocation failed. I'm not so confident we can do > > > > > that for all current consumers and both current and my proposed approach > > > > > don't impose such requirement. > > > > Cleanup should be identical to the actions done on close(2). > > > > > > > > > > > > > > Of course postponing fd allocation where possible is definitely worth > > > > > doing. > > > > Yes, and after that the rest of the cases should be evaluated. > > > > But my gut feeling is that everything would be converted. > > > > > > > > > > So let's say you accept() a connection. > > > > > > With current code, if you got to accept the connection you got it. > > > > > > With your proposal you may find that you can't allocate any fd and have > > > to close fp. This will be visible as accept + close by the other end, > > > while the caller never saw the connection. > > > > > > My guess is people would complain once they encounter such issue. > > > > Can't you already get this if you overflow the listen queue? (Having > > "accepted" connections aborted where the user application doesn't know): > > > > } else { > > /* > > * Keep removing sockets from the head until there's room for > > * us to insert on the tail. In pre-locking revisions, this > > * was a simple if(), but as we could be racing with other > > * threads and soabort() requires dropping locks, we must > > * loop waiting for the condition to be true. > > */ > > while (head->so_incqlen > head->so_qlimit) { > > struct socket *sp; > > sp = TAILQ_FIRST(&head->so_incomp); > > TAILQ_REMOVE(&head->so_incomp, sp, so_list); > > head->so_incqlen--; > > sp->so_qstate &= ~SQ_INCOMP; > > sp->so_head = NULL; > > ACCEPT_UNLOCK(); > > soabort(sp); > > ACCEPT_LOCK(); > > } > > TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list); > > so->so_qstate |= SQ_INCOMP; > > head->so_incqlen++; > > > > I think the simplest approach would be to first convert as many places as > > possible to use falloc_noinstall() / finstall(). If you end up with all of > > them converted then you can just rename falloc_noinstall to falloc() and > > retire the old falloc(). > > > > I would expect soabort to result in a timeout/reset as opposed to regular > connection close. > > Comments around soabort suggest it should not be used as a replacement > for close, but maybe this is largely because of what the other end will > see. That will need to be investigated. > > That said, I definitely support using delayed fd allocation (current > falloc_noinstall) where possible, but I'm not convinced it is safe for > all consumers. I think falloc_noinstall() failing is a rare case and is already going to cause application-level breakage that users have to handle anyway, so I'm not sure it's worth a lot of extra effort to support this edge case more gracefully. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:13:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3435BC23 for ; Thu, 14 Aug 2014 16:13:30 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8B4D2B03 for ; Thu, 14 Aug 2014 16:13:29 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id l13so14059021iga.4 for ; Thu, 14 Aug 2014 09:13:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=le4Q5q/klF/D/+OLRdK5XUcD/ZYpZMGgzhUYlwa1d0Y=; b=kFzO/wuiIHMNLAmg53ymmzbXa5xmGndQrv0E6h2fSoC+uZzmNSwKy9uqJxQkUM0MOG 86wldFs/mWTGP5gZgiaeXejto78rsW+rcPe5xeQFQvNc6Qx6gDYU4ZPTQoERQQc36OrJ XcgOWP0WbMxBXX65Q5zJ0xn1lUkVP0cSE/ZVecHNRnt52Yb2uJnLeeP8Fj6pU6prfy+v vA5wSFnGC5fvl9BH5Yn6Lpl+B4ngbAWx97Ahj9g2jiTdBrWRAhTr7bgnPg8cJx4LoEaH xaQZR7SfSuj6fHDODRf1LGN9y8mHChOz6qH6y6gUJY/vNARccCxZLHmROgXGL8PauOPx Byqg== X-Gm-Message-State: ALoCoQmFOk/uvuku7HCBfm/PvtMP3iOfUZa9/Gnl7GltHeZgBsHgfhCvqGjN+DbY4O1vF6KOrrBj X-Received: by 10.42.26.74 with SMTP id e10mr15247060icc.25.1408032808733; Thu, 14 Aug 2014 09:13:28 -0700 (PDT) Received: from netflix-mac.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id ro8sm21739789igb.15.2014.08.14.09.13.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 14 Aug 2014 09:13:27 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_DFC22E69-B2C2-435F-A022-562FF05965DC"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Warner Losh In-Reply-To: <201408140847.00573.jhb@freebsd.org> Date: Thu, 14 Aug 2014 10:13:26 -0600 Message-Id: <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> <201408140847.00573.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:13:30 -0000 --Apple-Mail=_DFC22E69-B2C2-435F-A022-562FF05965DC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Aug 14, 2014, at 6:47 AM, John Baldwin wrote: >>> Marking the binary with a libxo-specific note tells the caller that >>> the binary is capable of rendering its output in a non-traditional >>> style and gives the caller a means of triggering those styles of >>> output. In the libxo-enabled world, I see this as vital information >>> the caller needs to initialize the environment in which the command >>> will be run. Isn't this exactly the sort of information ELF targets >>> for note sections? >>=20 >> How binary format has any relevance for an application level feature = ? >> What would you do with the binaries which permissions are = 'r-s--x--x', >> which is not unexpected for the tools which gather system information >> and have to access things like /dev/mem ? >>=20 >> You removed and did not answered a crusial question, which is a = litmus >> test for your proposal. Namely, how presence of the proposed note in >> the binary is different from DT_NEEDED tag for your library ? >=20 > Yes, checking DT_NEEDED for libxo.so is the first thing I thought of = as well. =20 > It is equivalent to 'ldd foo | grep libxo'. Doesn=92t work for static binaries, nor for cases where libxo is linked = in by a library indirectly, nor for when the command is a shell script that may invoke a command that supports this output, nor for a python script that implements this output, etc. My question for people advocating this method: Why not require all = commands that generate this kind of output to support a standard command line = option that causes the command to print nothing and return 0 if it supports = reporting, or anything else if it doesn=92t (return 0 with output, or return = non-zero with or without output). This would handle the more complicated implementation issues = with using DT_NEEDED and/or the ELF note, be more in line with how things are = traditionally done, and offer greater flexibility of implementation. Warner --Apple-Mail=_DFC22E69-B2C2-435F-A022-562FF05965DC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJT7OAmAAoJEGwc0Sh9sBEAEPQQAJLliSBIX1ZOzojPmD3OMQjX JydNjyQxCL/8e9nwp3DbAJT3jdSuJCLNaPcyUr+9+bq49NNCGhHunyE6ORTewJRB Mn59lTRdrjNs1xFB8ZWjBuJgiWdMIyu14z9TJChm7x2wiTIjxW0bDJVfWlaGfVSU iYF+v039ethCGvHBEIFrX4nuovIqehMrcen1nnSd3MjjMSm3Q4/BZQpANpfmWVwN R2yjZkFG2YbfdxmlpRoAIMePHHcpAO3wD3UYTV0kcjHixFtkvJ3fGcdG6cowV17Q E9b3UkjbqrMmRuR+O+BzIBKepFzNmrSfmawwXxnGTU8wOhpWkSEaO6y0egtkNPCw KFYyaTspL0DjEY2jfsEw0sPqfMiexMV6qLivHQ+vNMFNJZIcE20mCJ3EhTXyW9FC 7PWoneU4L3v/QjsPmIlfwgule7rq+nuucCMPBFM4wzvegL7fZfr7gHw1VrUTScIy lKiQkD4Th+CTzGpmdj2p9EdmN9Us9Q7ZSBrTji8VyfDB+lIjtMcBzsaO6MeXiQjV Uh6M3O6O4yl8/PkTGbohQhyugJeTqN5S1hc9Y0yl+ui0q868WbW77LcSjU0DAV23 ou726heAes/0NuYgCdnwY8skV1lTX8CUVDhsCOy4x+1Fp725RPniafRr2MlqAgYl +04+85NTYfZOtHUZKmou =e42H -----END PGP SIGNATURE----- --Apple-Mail=_DFC22E69-B2C2-435F-A022-562FF05965DC-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:41:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4865FA6E; Thu, 14 Aug 2014 16:41:01 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0189.outbound.protection.outlook.com [207.46.163.189]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7CB2E2EE4; Thu, 14 Aug 2014 16:40:59 +0000 (UTC) Received: from BL2PR05CA0012.namprd05.prod.outlook.com (10.255.226.12) by DM2PR05MB735.namprd05.prod.outlook.com (10.141.178.24) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 14 Aug 2014 16:40:52 +0000 Received: from BN1AFFO11FD057.protection.gbl (2a01:111:f400:7c10::172) by BL2PR05CA0012.outlook.office365.com (2a01:111:e400:c04::12) with Microsoft SMTP Server (TLS) id 15.0.1010.18 via Frontend Transport; Thu, 14 Aug 2014 16:40:51 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1AFFO11FD057.mail.protection.outlook.com (10.58.53.72) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Thu, 14 Aug 2014 16:40:51 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 14 Aug 2014 09:40:50 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7EGeDn46015; Thu, 14 Aug 2014 09:40:26 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7EGe422096656; Thu, 14 Aug 2014 12:40:04 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408141640.s7EGe422096656@idle.juniper.net> To: Warner Losh Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> Date: Thu, 14 Aug 2014 12:40:04 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199003)(189002)(164054003)(81156004)(92566001)(21056001)(107046002)(99396002)(92726001)(85852003)(68736004)(4396001)(46102001)(6806004)(80022001)(86362001)(50986999)(74662001)(81542001)(81342001)(103666002)(76482001)(69596002)(74502001)(54356999)(84676001)(95666004)(64706001)(110136001)(53416004)(50466002)(20776003)(87936001)(83072002)(97736001)(47776003)(31966008)(79102001)(106466001)(48376002)(76506005)(77982001)(44976005)(85306004)(102836001)(83322001)(105596002); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB735; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 03030B9493 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:41:01 -0000 Warner Losh writes: >My question for people advocating this method: Why not require all >commands that generate this kind of output to support a standard >command line option that causes the command to print nothing and >return 0 if it supports reporting, or anything else if it doesn't >(return 0 with output, or return non-zero with or without output). It's a chicken and egg problem. I can't call the command with the option until I know that command can handle the option without generating an error, a core file, or rebooting the box. Until I know what the command will do, I can't invoke it safely. There's also the issue of find an option that all commands are not using, given that I can't change options for existing commands. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:50:05 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8FF17E0A for ; Thu, 14 Aug 2014 16:50:05 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 534B5201C for ; Thu, 14 Aug 2014 16:50:05 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id l13so14165715iga.4 for ; Thu, 14 Aug 2014 09:50:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=KbwhyFAQUB3rKs3mJo1iwYXzJGYbAtSlGxGnGPc88TU=; b=RJwW1eqWo1MpkVEtjlYF/VRwO4eiQ5hPoDDB4bO3DfA1KfHPb1vOd9tCglkRQ6qX1e YYf2Xn0TC+tjm9j0F/hL+AaEScXMeGMlUzv8fUdhoVXK99ne0j2rgzC2tWDtbnu4L6Ce T4yCzQNr6BKtFwbtIQsOAeUI3IiYDURH3ymc0V83SqrYMBT9ZR17ox9zR1C4c61swJ1S E1DcznYdJbfWLtz6VslWQqDABYhYLoUrsDpMQGdclAEC2Y1HcqPa/FkndDn7pxVtlPzC HvxfG8ebvfcLgZ7H53WOQxXVaYAX67U62QG4obXEk/pTawQavhPt7JK/EvGhQko1VC/K TjpA== X-Gm-Message-State: ALoCoQlBn7L5PLzTjfVJYEYGwA8GlxSHDw/HI0yfzd1K4yAM5OiIVZihR+8vcAo6HeiMZHRRPdpT X-Received: by 10.42.226.69 with SMTP id iv5mr16137178icb.43.1408035004750; Thu, 14 Aug 2014 09:50:04 -0700 (PDT) Received: from netflix-mac.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id hu9sm86307875igb.22.2014.08.14.09.50.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 14 Aug 2014 09:50:03 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_7B8A5A89-04AD-47AD-AF26-6952FEB25191"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Warner Losh In-Reply-To: <201408141640.s7EGe422096656@idle.juniper.net> Date: Thu, 14 Aug 2014 10:50:02 -0600 Message-Id: References: <201408141640.s7EGe422096656@idle.juniper.net> To: Phil Shafer X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:50:05 -0000 --Apple-Mail=_7B8A5A89-04AD-47AD-AF26-6952FEB25191 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Aug 14, 2014, at 10:40 AM, Phil Shafer wrote: > Warner Losh writes: >> My question for people advocating this method: Why not require all >> commands that generate this kind of output to support a standard >> command line option that causes the command to print nothing and >> return 0 if it supports reporting, or anything else if it doesn't >> (return 0 with output, or return non-zero with or without output). >=20 > It's a chicken and egg problem. I can't call the command with the > option until I know that command can handle the option without > generating an error, a core file, or rebooting the box. Until I > know what the command will do, I can't invoke it safely. If a userland command reboots the box in response to bad command line options, that=92s not your problem to fix: that=92s a security = issue that needs to be fixed regardless of the method you chose. If the = command creates a core file, that=92s a bug in that command. The command could very easily create a core file when you call it with a valid set of options too. Generating an error is 100% fine: in fact I count on that happening. > There's also the issue of find an option that all commands are not > using, given that I can't change options for existing commands. In my opinion, there=92s no chicken and egg problem. I specifically proposed a long option that isn=92t present in any = command, and is likely to generate errors or at least output. The reason I = proposed the long option was so that 2 lines of code could be added to programs that support it: in some header: #define LONG_OPTION_DEFINE =93=97supports-xml-output" #include ... int main(int argc, char **argv) { /* local variables here */ if (argc =3D=3D 2 && strcmp(argv[1], LONG_OPTION_DEFINE) =3D=3D = 0) exit(0); } Which wouldn=92t interfere with any other command line parsing these programs do. =97supports-xml-output isn=92t likely a valid set of = options for any program that exists today, except for those that support xml. The protocol is simple: redirect stdout and stderr to pipes, invoke the command, if the exit status is 0 and there=92s no output on the = pipes, then the command supports your protocol. If there=92s output, or if the exit status isn=92t 0, then the program doesn=92t. Warner --Apple-Mail=_7B8A5A89-04AD-47AD-AF26-6952FEB25191 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJT7Oi6AAoJEGwc0Sh9sBEAYlUQANQHwNrH68xkotZi27YYQthA ETaicmy1rterMVV16yoPAUVYxj3fNtMFiajlbDhQtbopPu0HYLJHq6aH4Y6lTVzm uQSfsSnlMbQthiaN0TdalkWKXar+hTLCz5TXYMyofvfV1NLELvzvHyq1C1mXw4NW 0V7E+5fouZxc/SBgUFace8ZMKbMRL9PSj1dnI96zMQ28WwCeq4HbH297tVwV1sRk OmolUKJSONAczauPdoS9t7v5SsRPhVCfBE7Znm0UwEyKUEcKuTYd/VdFOxQw5Mkg 3ThYGHlgoJKNC3ztIcUxGqSUMJrQ3OUxcArzwXh/MGH/hQbz+01bk37ZXO95TvzT aP7NKDot0Yc9wf2R/J6uo21gflQCSBq94ZKJUxJbZY5ZyWDpWtaKrKsh8jvvb2j5 vzKQYmUFLsYuM9BhJVnXQryzgAL3geB4GPCkAmd8NfNFp2vvMAgNygjzDpxWTUzN uggBj5mxfX1zowKqi4DPqpHjSWpsF5PIH1GNv+KWDEkDtfj0UcOHM9dX2LFR939R uNzdPwRpDSonbfrMVzkOAOJtqTTgqq2DpjsCA2+5zF1k+/k09GU6zcSiDNdtNBeM K7pq7/ZlwqkuvXmfzV+++mzS0I4B0JAOFIHNGgK3rGBDM4eqfCJJYJaaaiBZeSGj ks1FWXpH2YXW0BHLdcMR =RWuk -----END PGP SIGNATURE----- --Apple-Mail=_7B8A5A89-04AD-47AD-AF26-6952FEB25191-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 16:55:14 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF701A2; Thu, 14 Aug 2014 16:55:14 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BEDFD20D9; Thu, 14 Aug 2014 16:55:14 +0000 (UTC) Received: from rcannon-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s7EGtABd003818 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 14 Aug 2014 09:55:11 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_90C93917-7538-4938-B778-D1BD0878691D"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <201408140452.s7E4qB2X091511@idle.juniper.net> Date: Thu, 14 Aug 2014 09:55:04 -0700 Message-Id: <39394555-F1DE-4D8E-9161-AD179EB3B8FC@xcllnt.net> References: <201408140452.s7E4qB2X091511@idle.juniper.net> To: Phil Shafer X-Mailer: Apple Mail (2.1878.6) Cc: arch@freebsd.org, Poul-Henning Kamp , Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:55:15 -0000 --Apple-Mail=_90C93917-7538-4938-B778-D1BD0878691D Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Aug 13, 2014, at 9:52 PM, Phil Shafer wrote: > >> The question is: do we have >> more features we want to check for this way? If not, then >> such a scheme could be perceived as "heavy handed". > > "heavy handed" in what sense? A generic implementation calls for a utility that can list features, check for a feature, add and remove features. You probably want to have support libraries as well to do the same thing from within C and C++, etc. If all we're talking about is libxo then there are surely less work-intensives ways to achive what we want. In that sense heavy-handed. > I'm hoping the note can be added by normal linker magic (but see > question below). If not a "noteelf" command would need to be created > (or an option to brandelf?) to mark the binary. Ideally, features set in the same way we do for. the kernel. Only in the kernel it ends up being a sysctl, which doesn't apply here. Alternatives in the kernel are linker sets and I've used those in mkimg as well. A linker set is like a note, except that the metadata is in a loadable section and thus part of a loaded segment. This has the advantage of being able to go over your own set of features from within C and C++, etc and it doesn't require a utility to add and remove features to a set. Checking for a feature from some other program is probably as hard as a note, assuming there was a single note to list all features. In both cases you first have to find the container with some ELF grokking utility and then you need to parse the container's data to find a feature. A note has the benefit over a section in that I can make changes without having (effectively) to relink. > Are you seeing > something more? "elfdump" has a "-n" to dump notes, and a "-N > " could be added, making "elfdump -n -N libxo my-app" the > means of getting the contents. A "-q" option could be added to > prevent output but set the exit code based on if the section appears > in the given binary. I think I prefer a more intergated solution in the end. elfdump is that "ELF grokking" utility, but may not necessarily be appropriate for the interpretation of the feature set. But that's for later. For now, something that works as a prototype is good enough. > >> Alternatives include looking for a particular symbol or >> possibly even running the utility with a libxo option that >> has predictable output. > > How does one put a symbol in a binary when linking against a shared > library? One doesn't necessarily. A recursive search that follows the DT_NEEDED could possibly work well enough. Though, I fear for the likes of Mozilla that large number of shared libraries. Not that such is our immediate scope/target, but if there is an approach that scales well, then that would be better. > Would there need to be two libs, one with the code and > one with just a symbol? I'd have the same issue with the ElfNote > scheme, right? I'd need to add a section to the binary, but libxo > could be linked dynamically. Is there an easy answer for this? Or > is the app stuck with "LDADD+=-lxo -lxo-note"? I don't have a good answer other than that there isn't a suitable option to LD to leave a turd in the executable. The absolute easiest is to simply not support static linking for libxo and then simply check DT_NEEDED. You can use ldd for that. That route wouldn't require anything other than not (ever) building an archive library for libxo. This is not unreasonable either. The world has moved away from archive linking for the most part and while there's still value to static linking, I see it needed less and less. -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_90C93917-7538-4938-B778-D1BD0878691D Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPs6egACgkQpgWlLWHuifbmsACdEl1/+hyNwUkBG+DfL2r2iOHi 840AnAzX8854JbJ89y53tifxXsaZwQYZ =0Ysq -----END PGP SIGNATURE----- --Apple-Mail=_90C93917-7538-4938-B778-D1BD0878691D-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 17:04:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E63D34F for ; Thu, 14 Aug 2014 17:04:54 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB53E21CF for ; Thu, 14 Aug 2014 17:04:53 +0000 (UTC) Received: from rcannon-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s7EH4gCt003857 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 14 Aug 2014 10:04:44 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_3EF02EE7-3339-4832-ACFA-93F4CF617EF9"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <20140814052648.GM2737@kib.kiev.ua> Date: Thu, 14 Aug 2014 10:04:36 -0700 Message-Id: <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 17:04:54 -0000 --Apple-Mail=_3EF02EE7-3339-4832-ACFA-93F4CF617EF9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Aug 13, 2014, at 10:26 PM, Konstantin Belousov = wrote: >>=20 >> I know ELF "Note" elements can be used to carry vendor-specific >> data, but have no experience with them. Would it be reasonable to >> use them as a means of communicating this information to other bits >> of software? > No. Too extreme. >> Is FreeBSD using Notes for other information currently? > Yes, the notes are used to communicate the information required by > the dynamic linker to correctly activate the image. The mechanism has > nothing to do with application-specific features, and overloading it = for > that purpose is severe and pointless layering violation. Things should > not be done just because they could be done. Too extreme. Life is a lot more subtle. Standards are as well. There are many examples in the real world where standards are interpreted a little more liberal than others may want to. When such result in (gratuitous) incompatibilities, we all interpret it as bad. But when it adds real value, you tend to find it in the next update of the standard. > Using the static tagging for the dynamic application properties is = wrong > anyway. E.g., would you consider the mere fact that the binary is = linked > against your library, as the indication that your feature is supported = ? > If not, how does it differ from the presence of some additional note ? If we can eliminate static linking for libxo, than that is definitely easy. Easiest probably. The question becomes: is it acceptable to not support static linking for libxo? Or alternatively, is it acceptable to not be able to check for the feature on a static executable? For the first I'm inclined to say yes, but not for the second. --=20 Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_3EF02EE7-3339-4832-ACFA-93F4CF617EF9 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPs7CQACgkQpgWlLWHuifYLbwCeLctj2ViFfNYa5pLIvwmgpqeE B4IAnAvxZQ3XPTT7KGd++1ID+MPv21ko =LXa8 -----END PGP SIGNATURE----- --Apple-Mail=_3EF02EE7-3339-4832-ACFA-93F4CF617EF9-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 17:08:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 84EFE438 for ; Thu, 14 Aug 2014 17:08:01 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3F24321F7 for ; Thu, 14 Aug 2014 17:08:01 +0000 (UTC) Received: from rcannon-sslvpn-nc.jnpr.net ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s7EH7m6x003871 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 14 Aug 2014 10:07:49 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_46B2DA2C-D1E8-481B-9BE3-2CD8865D8992"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Marcel Moolenaar In-Reply-To: <20140814085257.GN2737@kib.kiev.ua> Date: Thu, 14 Aug 2014 10:07:42 -0700 Message-Id: References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 17:08:01 -0000 --Apple-Mail=_46B2DA2C-D1E8-481B-9BE3-2CD8865D8992 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Aug 14, 2014, at 1:52 AM, Konstantin Belousov wrote: > On Thu, Aug 14, The ELF spec says: >> >> >> Note Section >> >> Sometimes a vendor or system builder needs to mark an object >> file with special information that other programs will check >> for conformance, compatibility, etc. Sections of type SHT_NOTE >> and program header elements of type PT_NOTE can be used for >> this purpose. The note information in sections and program >> header elements holds any number of entries, each of which is >> an array of 4-byte words in the format of the target processor. >> Labels appear below to help explain note information organization, >> but they are not part of the specification. > ELF standard scope is about build toolchain and C runtime, where the > cited paragraph makes perfect sense. That's a self-imposed precondition that is not present in the spec. You're just as liberal in the interpretation as Phil is. You just on the other side of the argument. I am definitely interested in what you think a system builder is given your objection. -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_46B2DA2C-D1E8-481B-9BE3-2CD8865D8992 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPs7N4ACgkQpgWlLWHuifY1aQCeI5zBeGuTHnDM2jo3UyGiOILv X1IAnA8MlaBwZX1wDup6twIkJBs+JKLx =nsd3 -----END PGP SIGNATURE----- --Apple-Mail=_46B2DA2C-D1E8-481B-9BE3-2CD8865D8992-- From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 17:12:15 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D27276B4; Thu, 14 Aug 2014 17:12:15 +0000 (UTC) Received: from mail-qa0-x232.google.com (mail-qa0-x232.google.com [IPv6:2607:f8b0:400d:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8347A22A3; Thu, 14 Aug 2014 17:12:15 +0000 (UTC) Received: by mail-qa0-f50.google.com with SMTP id s7so1175032qap.23 for ; Thu, 14 Aug 2014 10:12:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6j+A+oW91Zgydj4KVISrAllRS1Rv6k8lUBS9fR0cSpc=; b=tRZmjaQB8fsyqEZ8gtX46b+z26UXRRIt+0fziULcr3YHUsneGDiyDkGWC22AACv25j G5nK3qwqejQSVQr76FYBB7w00AXwnISBL7lby5KNQwW1BxzI+3+e22fnWKtFdg8r1YNX BKUULgPKBdMetkQEDsZpQ5PkFEgvJGnHCjy3ATDxohzDLKZ694HgYH/5i5ARfGqD5e+A /vZ4mAYn1Pq0E00DWx/NMfl41Eb/YcCLeJhVQ9Qh5NbIjAFTgcSptSiVoEtOTpvO6nul B34svEHBNeCvOJsRdm3X/iGxaMjLt0uyX6xVtiv/qaX5l0YHSQnBEzZ1tmJ5+EZl38iU CjbA== MIME-Version: 1.0 X-Received: by 10.224.30.77 with SMTP id t13mr16616092qac.63.1408036333641; Thu, 14 Aug 2014 10:12:13 -0700 (PDT) Received: by 10.224.39.139 with HTTP; Thu, 14 Aug 2014 10:12:13 -0700 (PDT) In-Reply-To: <201408141143.01137.jhb@freebsd.org> References: <201408120939.56176.jhb@freebsd.org> <201408141143.01137.jhb@freebsd.org> Date: Thu, 14 Aug 2014 10:12:13 -0700 Message-ID: Subject: Re: RFC: cpuid_t From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 17:12:16 -0000 On 14 August 2014 08:43, John Baldwin wrote: > On Tuesday, August 12, 2014 6:17:27 pm Adrian Chadd wrote: >> On 12 August 2014 06:39, John Baldwin wrote: >> >> > I think Bruce's point is you can just fix the few places that use a u_char to >> > an int and call it a day. The biggest offender of those that I can think of >> > are td_oncpu/td_lastcpu and the corresponding fields in kinfo along with >> > expanding NOCPU to be -1 instead of 255. I think most other places already >> > use an int (pf is one place that doesn't, it borrows N bits from some other >> > field to store the CPU number, so it can't use a cpuid_t anyway), so the patch >> > to just use an int might actually be quite small. >> >> You could do that, but then we'd be left in a situation that it isn't >> 100% clear what type people should be using (except hoping they >> cut/paste code that uses an int and don't make dumb decisions like >> assuming they can pack it into other fields, like pf) and that they >> could treat the CPU ID as some kind of index number. Being able to >> grep for all the places where cpuid_t is will make it a lot easier in >> the future to uplift all of the cpu id manipulating code to treat CPU >> IDs differently (eg making things very sparse.) > > I'm not sure how a typedef makes it easier to handle sparse IDs over an int? > Plus, the IDs are already allowed to be sparse (hence CPU_ABSENT() and > CPU_FOREACH(), etc.). I doubt that a typedef would make it any easier to > deal with issues like pf either because the current code is not explicitly > using any type at all, it's just passed into magic macros. It doesn't. It's there to help you find places where people aren't doing the iteration correctly. You can do tricks like making cpuid_t temporarily a struct with the int embedded in it: it'll work fine where it's being copied around as an opaque type, but it'll compiler error out the minute some code tries passing it into a field that takes an int, or a char. >> So, I'm happy if the group decision is to just make it an int. It just >> has to be done before FreeBSD-11 ships. :) > > I think making it an int would be a good first step at the very least. It > would be good to see how many places don't already use it that way. > >> I don't have such an aversion to typedefs. It has always made it much >> easier to do type checking and changes to things in C that I'd >> normally do in C++ with classes. > > One downside is that for printf() (a non-trivial thing in C), you have to > either violate the abstraction (i.e. "know" it's an int) or use intmax_t casts > everywhere. That's why you have a cpuid_t_to_quad() macro and use that everywhere you need a numerical representation of the ID. That way it's up to the macro or inline to return it as the type you require and thus printf() and friends can have a well-known type in the string. (I dislike intmax_t casts. I'd rather there were explicit ways to convert those types to integer representations where appropriate rather than having the code do those casts just to print. Ugh.) -a From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 17:17:16 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F3DE7895; Thu, 14 Aug 2014 17:17:15 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (dns-bn1lp0143.outbound.protection.outlook.com [207.46.163.143]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB6E722D9; Thu, 14 Aug 2014 17:17:14 +0000 (UTC) Received: from BLUPR05MB724.namprd05.prod.outlook.com (10.141.207.154) by BLUPR05MB628.namprd05.prod.outlook.com (10.141.204.156) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Thu, 14 Aug 2014 17:17:06 +0000 Received: from BN1PR05CA010.namprd05.prod.outlook.com (10.255.197.30) by BLUPR05MB724.namprd05.prod.outlook.com (10.141.207.154) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Thu, 14 Aug 2014 17:17:05 +0000 Received: from BN1AFFO11FD027.protection.gbl (2a01:111:f400:7c10::159) by BN1PR05CA010.outlook.office365.com (2a01:111:e400:400::30) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Thu, 14 Aug 2014 17:17:04 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1AFFO11FD027.mail.protection.outlook.com (10.58.52.87) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Thu, 14 Aug 2014 17:17:04 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 14 Aug 2014 10:16:52 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7EHGln70544; Thu, 14 Aug 2014 10:16:48 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 2241D580A2; Thu, 14 Aug 2014 10:16:47 -0700 (PDT) To: Warner Losh Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> <201408140847.00573.jhb@freebsd.org> <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> Comments: In-reply-to: Warner Losh message dated "Thu, 14 Aug 2014 10:13:26 -0600." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 14 Aug 2014 10:16:47 -0700 Message-ID: <20140814171647.2241D580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199003)(99396002)(31966008)(105596002)(79102001)(80022001)(74502001)(85852003)(4396001)(77982001)(81342001)(90896003)(50226001)(86362001)(50466002)(93886004)(62966002)(87936001)(46102001)(69596002)(33656002)(81156004)(95666004)(85306004)(104166001)(76506005)(48376002)(92566001)(64706001)(68736004)(83322001)(102836001)(21056001)(97736001)(101356003)(76482001)(47776003)(76176999)(87286001)(89996001)(70486001)(93916002)(57986006)(74662001)(88136002)(110136001)(102176002)(77156001)(84676001)(93546004)(83072002)(20776003)(50986999)(106466001)(44976005)(81542001)(6806004)(107046002)(92726001)(42262002); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB724; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:;UriScan:; X-Forefront-PRVS: 03030B9493 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 17:17:16 -0000 >My question for people advocating this method: Why not require all = >commands >that generate this kind of output to support a standard command line = >option That's basically what we did in Junos, except the -X option currently just means output XML. That worked fine for the limited number of bsd apps that we frobbed, but pretty hard to assert that it could be expanded to all apps. Thus I think Phil was looking for a more generic solution. I think your suggestion could work fine. Since (I think) Phil mentioned libxo being able to check options, it should be ok to run an app with an option like: --libxo-is-supported that's guaranteed to not conflict with any existing option. Any app that hasn't been converted will choke and die, any that has will exit happy. You could add other --libxo-* options to control output - if environment variables are not considered desirable From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 18:31:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABD9AF2A; Thu, 14 Aug 2014 18:31:30 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FC172CBB; Thu, 14 Aug 2014 18:31:30 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 79C70B987; Thu, 14 Aug 2014 14:31:28 -0400 (EDT) From: John Baldwin To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Thu, 14 Aug 2014 14:22:08 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140814052648.GM2737@kib.kiev.ua> <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> <20140814171647.2241D580A2@chaos.jnpr.net> In-Reply-To: <20140814171647.2241D580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201408141422.08265.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 14:31:28 -0400 (EDT) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 18:31:30 -0000 On Thursday, August 14, 2014 1:16:47 pm Simon J. Gerraty wrote: > > >My question for people advocating this method: Why not require all = > >commands > >that generate this kind of output to support a standard command line = > >option > > That's basically what we did in Junos, except the -X option currently > just means output XML. > That worked fine for the limited number of bsd apps that we frobbed, > but pretty hard to assert that it could be expanded to all apps. > > Thus I think Phil was looking for a more generic solution. > > I think your suggestion could work fine. > Since (I think) Phil mentioned libxo being able to check options, > it should be ok to run an app with an option like: > > --libxo-is-supported > > that's guaranteed to not conflict with any existing option. > Any app that hasn't been converted will choke and die, any that has will > exit happy. > > You could add other --libxo-* options to control output - if environment > variables are not considered desirable I vote for this. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 18:36:20 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 06A6112E for ; Thu, 14 Aug 2014 18:36:20 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D60772CF0 for ; Thu, 14 Aug 2014 18:36:19 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id DE8EBB922 for ; Thu, 14 Aug 2014 14:36:18 -0400 (EDT) From: John Baldwin To: arch@freebsd.org Subject: Bumping cpuset_t size and MAXCPU on amd64 to 256 Date: Thu, 14 Aug 2014 14:35:59 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201408141435.59323.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 14:36:19 -0400 (EDT) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 18:36:20 -0000 There was an earlier thread about this a few months ago and after fixing a few other places (already committed) the patch below has survived a universe build. Does anyone have any objections to these going in? Note that the _cpuset.h change changes the size of 'cpuset_t' in userland for all platforms. Due to the way the cpuset system calls work (and the libc routines), this should not be an ABI change (the caller always encodes the size of the set as an argument to the system call). Also, note that on amd64 the kernel won't actually handle more than 255 CPUs (a few places still use u_char which Adrian is preparing to beat into submission, but we also can't support APIC IDs > 0xfe until we grow true x2APIC support). Index: sys/amd64/include/param.h =================================================================== --- sys/amd64/include/param.h (revision 269991) +++ sys/amd64/include/param.h (working copy) @@ -65,7 +65,7 @@ #if defined(SMP) || defined(KLD_MODULE) #ifndef MAXCPU -#define MAXCPU 64 +#define MAXCPU 256 #endif #else #define MAXCPU 1 Index: sys/sys/_cpuset.h =================================================================== --- sys/sys/_cpuset.h (revision 269991) +++ sys/sys/_cpuset.h (working copy) @@ -38,7 +38,7 @@ #define CPU_SETSIZE MAXCPU #endif -#define CPU_MAXSIZE 128 +#define CPU_MAXSIZE 256 #ifndef CPU_SETSIZE #define CPU_SETSIZE CPU_MAXSIZE -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Aug 14 23:22:54 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C399E4F for ; Thu, 14 Aug 2014 23:22:54 +0000 (UTC) Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62D71201C for ; Thu, 14 Aug 2014 23:22:53 +0000 (UTC) Received: by mail-ig0-f169.google.com with SMTP id r2so947307igi.0 for ; Thu, 14 Aug 2014 16:22:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:content-type:subject:message-id:date :to:mime-version; bh=Z3vkO9BUeCsY0IxK7dABhEgC6Fp4JXmGu/WUQjA1+ow=; b=QXfhb4WmRe+NRG1QSI9/tZl+jTBElESHcnhX5R4ky3FaLwpCN1UsKOsO+cYlNYfPeA H4aKWhso3dXmcSqDXcXHvahoUkZTKUEZ8CEvJmPzklZPvUaBe5g4YlpRhYz7rM9J3zV2 IEQjo/JwbcHymhoJKx4r367oavTFicHkyetS4T6LlZCoqMWXu2eeyN/34qucAE0auk0R 1NJgNm6WhYNx56glw1btWVSby8bijx7g+V3nDbXL+V4FvWmgVhHo2xT8cXpoJ/5LPwsZ ASNta+EQmm+aaGQDWVmZy35qgtkV8g+7tipW/Fgp0nNthA8fyuyEML6sT6YdOg2VCD9S wdOQ== X-Gm-Message-State: ALoCoQmH3qf9B/nxGr7TC2Bv58cqwJTA7v4v6mE54HvApOSj2LWxMosNIOfLR9bGC32350ZkWa5j X-Received: by 10.50.32.73 with SMTP id g9mr63907841igi.31.1408058572670; Thu, 14 Aug 2014 16:22:52 -0700 (PDT) Received: from netflix-mac.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id b8sm928530igl.5.2014.08.14.16.22.51 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 14 Aug 2014 16:22:52 -0700 (PDT) Sender: Warner Losh From: Warner Losh X-Google-Original-From: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_34F8677D-C137-4C06-A051-240664178A51"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: MD Elf header filtering Message-Id: Date: Thu, 14 Aug 2014 17:22:50 -0600 To: "freebsd-arch@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 23:22:54 -0000 --Apple-Mail=_34F8677D-C137-4C06-A051-240664178A51 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Greetings I=92ve implemented a per-arch filtering mechanism for elf headers. why = would you need this? Because we don=92t have a kernel that can run both = EABI and OABI binaries on arm. OABI binaries can lead to crashes when = run on an EABI kernel. To prevent that, I=92ve created a new function = elfXX_check_header_md() which interested architectures can implement to = filter out binaries they know won=92t work and use it to filter out EABI = binaries on an OABI kernel and vice versa on arm. I=92ve also created a = weak default version for those architectures that don=92t that accepts = everything... https://reviews.freebsd.org/D609 Comments? Warner --Apple-Mail=_34F8677D-C137-4C06-A051-240664178A51 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJT7UTLAAoJEGwc0Sh9sBEA4JIP/3BMgYuH5UXgmg5oBV1pPoZZ Kz3adbLtTNTlTfWzLJRtbT06SEEhoSgws4bwOWHl818YdA7yCS3i10mnLUQuC9lD nNm7ZHIHw5FB4aYNLD30kh0w275TnzoI2AD7hClQfsjlEQ+X2KJ+73SrSN64Bd/u Hyj5nhOIXQS6M+bnWhLW2BpCST+gFpMgRDo3wtnwb1ZqiCpQ969jK9UmSBLzpO1p KTFpkB1NZJK48Nk9IPDO5GY8OMyLnKpayDhZTHmpWeQGyEUByVpUT9vzVgSTFpj7 cA1G1KKG66gYckPOoM/Y7LwXgwUoEGoFfzvss3qV5G1H6zLip46WfhxgloJT8XOV Th/yd0ZQHuWDMGN4Emu0I+7ybTZGJGuuDosoIfhor58a9UmbKA0H5lsDDFkr3F9d jBkuvla85AGjA97bdH5UknjVOMKeFRxWvyOiaQUvnRNhSzTz0plI3e427ltKub+I s7/euSgYPW6UhLpSZ3xqmSZTmOtJhFXARwk6tHDHT1C0f3e1J1Tl5xt9zte5YYmz tI/4VMtMy8hDnYb/zrqrKLLco7mFlebBHVP8LeAKiMpx+y92D30BEaC2m1xYb5cf EWOFYKN8qCPQXicXCZyh535Jxy8zfMqn0RxKbzGfV7v3AzkY474qaZPcANNTX+82 MI/Dr+rAvblqtkfXTkfX =TWVK -----END PGP SIGNATURE----- --Apple-Mail=_34F8677D-C137-4C06-A051-240664178A51-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:39:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E0F9BCDF; Fri, 15 Aug 2014 00:39:30 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id CC1D127A6; Fri, 15 Aug 2014 00:39:30 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 72845346DE36; Thu, 14 Aug 2014 17:39:22 -0700 (PDT) Message-ID: <53ED571A.9030503@mu.org> Date: Thu, 14 Aug 2014 17:40:58 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: freebsd-arch@freebsd.org, John-Mark Gurney , marcel@freebsd.org, Poul-Henning Kamp , Konstantin Belousov , Phil Shafer , Warner Losh Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408141516.s7EFGE4a096197@idle.juniper.net> <746AC443-B255-47DD-8C24-3E3A32A5CC05@bsdimp.com> In-Reply-To: <746AC443-B255-47DD-8C24-3E3A32A5CC05@bsdimp.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:39:31 -0000 On 8/14/14 8:23 AM, Warner Losh wrote: > Sorry for top posting, this really isn=92t responsive to the minutia in= the rest of the thread. > > I=92m curious. Why isn=92t this conversation about =93foo =97supports-x= ml=94 ? Why tag these commands with weird, non-standard things that need = more exotic tools to dig the information out. Why not have a standardized= command line option that prints nothing and returns 0 for success, or wh= ines and returns 1 for failure? That=92s way more standardized than addin= g obscure notes that may or may not be allowed by the standard, but that = we traditionally haven=92t done, which requires tools that aren=92t stand= ardized and whose interface varies from one tool to the next. This is tru= e of asking about DT_NEEDED (which forces a specific library for the impl= ementation) as well as anything placed in the NOTES section. It also assu= mes that you know the thing you are querying is an ELF executable, that y= ou can find it, that there=92s not a shell script wrapper for that tool t= hat redirects to binaries that do support this, etc, etc etc. > > Basically, what does this =91meta data=92 really buy you that can=92t b= e bought some other, more standard, more direct way that doesn=92t enshri= ne so many hard-coded implementation decisions into the mix? In addition I am wondering what branding the binaries really offers "as-i= s". Example, let's say you have a means to query and find out that "netstat" = supports libxo. Well, netstat has many output variants: netstat netstat -r netstat -a netstat -nr netstat -na netstat -p tcp So given that is appears that we want to build something so that "file=20 browsers" can automatically determine that a program can be run in=20 "libxo" mode and some form of output should be rendered, what exactly is = the preferred format? What happens when a particular program's default behavior is to filter=20 stdin, but yet supports libxo, how is that handled? What happens when a particular program's default behavior is to run=20 indefinitely, and yet supports libxo, how is that handled? It makes sense to limit the scope of the project to just doing the=20 formatted output at least until we see what we get when get a whole=20 bunch of tools running with it. Speaking of getting a whole bunch of tools running with it, the GSOC=20 project happened to have near a dozen programs converted, how is the=20 libxo project coming along, do we have more programs converted? Without=20 the programs converted we don't have very much to show even with a great = library. What other apps use libxo in the tree now? -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:42:59 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10BA9D9F; Fri, 15 Aug 2014 00:42:59 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id EFB152836; Fri, 15 Aug 2014 00:42:58 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 5992E346DE46; Thu, 14 Aug 2014 17:42:58 -0700 (PDT) Message-ID: <53ED57F2.5020808@mu.org> Date: Thu, 14 Aug 2014 17:44:34 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Phil Shafer , Warner Losh Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408141640.s7EGe422096656@idle.juniper.net> In-Reply-To: <201408141640.s7EGe422096656@idle.juniper.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:42:59 -0000 On 8/14/14 9:40 AM, Phil Shafer wrote: > Warner Losh writes: >> My question for people advocating this method: Why not require all >> commands that generate this kind of output to support a standard >> command line option that causes the command to print nothing and >> return 0 if it supports reporting, or anything else if it doesn't >> (return 0 with output, or return non-zero with or without output). > It's a chicken and egg problem. I can't call the command with the > option until I know that command can handle the option without > generating an error, a core file, or rebooting the box. Until I > know what the command will do, I can't invoke it safely. > > There's also the issue of find an option that all commands are not > using, given that I can't change options for existing commands. > > I don't understand the need to query these programs for support of the option. Can you explain in more detail? -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:55:19 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17DA396; Fri, 15 Aug 2014 00:55:19 +0000 (UTC) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 38106290F; Fri, 15 Aug 2014 00:55:18 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id w62so1807079wes.15 for ; Thu, 14 Aug 2014 17:55:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6yC0IPWCG3qT4LJ9DTzfFywurlLVSsr7jPiw7ugjI2g=; b=VYeLey3N1+/B7bh8JwtEdLakxxNiRjUOnt1vi7PbBQDdMC9QVn9LrBDcUTgKT7Eiq6 2f8VYzV4jmdYEEqLXEiL3fZwF17vtaoeoBzqGve2XY2Ji1yX4Rg7/UJnpflU2y6R48yE eJ4LOldJ4e/Q+64nS/26IHJc52heudjM4TxV3dtUAZieWKZfzJOsv8zoDHbX1fGw/+NT DlohW4xG+IqQxdG/WZ4b39kj5gWHK5xye2CJOo6lqkhGB1L39Uh9CPOJbv15DODPxDr9 02aZmkmJ/B/89nfPuW2o4IL1+SxMj99c5IJymdhL0YK6lc4dCBYDWlRfrQl4HmSkYx0u v9vQ== X-Received: by 10.194.203.105 with SMTP id kp9mr16985375wjc.41.1408064116467; Thu, 14 Aug 2014 17:55:16 -0700 (PDT) Received: from localhost.localdomain (ip-62-245-66-51.net.upcbroadband.cz. [62.245.66.51]) by mx.google.com with ESMTPSA id fp6sm1473669wic.11.2014.08.14.17.55.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Aug 2014 17:55:15 -0700 (PDT) From: Mateusz Guzik To: freebsd-arch@freebsd.org Subject: [PATCH 1/2] Implement simple sequence counters with memory barriers. Date: Fri, 15 Aug 2014 02:55:11 +0200 Message-Id: <1408064112-573-2-git-send-email-mjguzik@gmail.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1408064112-573-1-git-send-email-mjguzik@gmail.com> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> Cc: Robert Watson , Johan Schuijt , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:55:19 -0000 --- sys/sys/seq.h | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 sys/sys/seq.h diff --git a/sys/sys/seq.h b/sys/sys/seq.h new file mode 100644 index 0000000..0971aef --- /dev/null +++ b/sys/sys/seq.h @@ -0,0 +1,126 @@ +/*- + * Copyright (c) 2014 The FreeBSD Project + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _SYS_SEQ_H_ +#define _SYS_SEQ_H_ + +#ifdef _KERNEL + +/* + * Typical usage: + * + * writers: + * lock_exclusive(&obj->lock); + * seq_write_begin(&obj->seq); + * ..... + * seq_write_end(&obj->seq); + * unlock_exclusive(&obj->unlock); + * + * readers: + * obj_t lobj; + * seq_t seq; + * + * for (;;) { + * seq = seq_read(&gobj->seq); + * lobj = gobj; + * if (seq_consistent(&gobj->seq, seq)) + * break; + * cpu_spinwait(); + * } + * foo(lobj); + */ + +typedef uint32_t seq_t; + +/* A hack to get MPASS macro */ +#include +#include + +#include + +static __inline bool +seq_in_modify(seq_t seqp) +{ + + return (seqp & 1); +} + +static __inline void +seq_write_begin(seq_t *seqp) +{ + + MPASS(!seq_in_modify(*seqp)); + (*seqp)++; + wmb(); +} + +static __inline void +seq_write_end(seq_t *seqp) +{ + + wmb(); + (*seqp)++; + MPASS(!seq_in_modify(*seqp)); +} + +static __inline seq_t +seq_read(seq_t *seqp) +{ + seq_t ret; + + for (;;) { + ret = READ_ONCE(*seqp); + if (seq_in_modify(ret)) { + cpu_spinwait(); + continue; + } + break; + } + + rmb(); + + return (ret); +} + +static __inline seq_t +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) +{ + + MPASS(!seq_in_modify(oldseqp)); + return (*seqp == oldseqp); +} + +static __inline seq_t +seq_consistent(seq_t *seqp, seq_t oldseqp) +{ + + rmb(); + return (seq_consistent_nomb(seqp, oldseqp)); +} + +#endif /* _KERNEL */ +#endif /* _SYS_SEQ_H_ */ -- 2.0.2 From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:55:20 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F5FC119; Fri, 15 Aug 2014 00:55:20 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96D4A2910; Fri, 15 Aug 2014 00:55:19 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id f8so238590wiw.0 for ; Thu, 14 Aug 2014 17:55:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OPFrezGku/0dhsMwF5pVJHfvVjPS6cvspl0KqB0pYBE=; b=Z+EP5uAOLazSRcJSPeI/28GZaxCNwl4cRF7xc91poRi1E7KVbAo6ECrLDNbXYwDbqQ XVamWUiKksd282yxvROdGM8BCEtvwyACkgWtuk9qs4/7xttbdFNBoRUWaFD//oKVKLVr tii+k9BOujTXHezFb1fYW7nwx02evOWGp6TB5iAUlSVEE9URm+GwgX0TIMsKoizGdztU iNzPwThVtgfhfUWFx3tBfxLEfiFA0qZM0cxrzkiFgaon1BlCMC2uvxnOwPVVJTEsX5r6 7W2dSU7yWftvgNQskfdp0jG/WUZwUPRPzsySMbgyjZZZ8Q71GmfZXlixfrkIISgRhWJ1 tQAg== X-Received: by 10.194.237.5 with SMTP id uy5mr15017160wjc.98.1408064117991; Thu, 14 Aug 2014 17:55:17 -0700 (PDT) Received: from localhost.localdomain (ip-62-245-66-51.net.upcbroadband.cz. [62.245.66.51]) by mx.google.com with ESMTPSA id fp6sm1473669wic.11.2014.08.14.17.55.16 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Aug 2014 17:55:17 -0700 (PDT) From: Mateusz Guzik To: freebsd-arch@freebsd.org Subject: [PATCH 2/2] Use sequence counters to make sure fget_unlocked gets a file pointers in sync with its capabilities. Date: Fri, 15 Aug 2014 02:55:12 +0200 Message-Id: <1408064112-573-3-git-send-email-mjguzik@gmail.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1408064112-573-1-git-send-email-mjguzik@gmail.com> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> Cc: Robert Watson , Johan Schuijt , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:55:20 -0000 Otherwise there are exploitable races (e.g. with dup2) which can be used to circumvent protection. --- sys/kern/kern_descrip.c | 55 ++++++++++++++++++++++++++++++++++++++++++------- sys/sys/filedesc.h | 16 ++++++++++++++ 2 files changed, 64 insertions(+), 7 deletions(-) diff --git a/sys/kern/kern_descrip.c b/sys/kern/kern_descrip.c index 4aeb380..1033b2f 100644 --- a/sys/kern/kern_descrip.c +++ b/sys/kern/kern_descrip.c @@ -309,11 +309,18 @@ _fdfree(struct filedesc *fdp, int fd, int last) struct filedescent *fde; fde = &fdp->fd_ofiles[fd]; +#ifdef CAPABILITIES + if (!last) + seq_write_begin(&fde->fde_seq); +#endif filecaps_free(&fde->fde_caps); if (last) return; - bzero(fde, sizeof(*fde)); + bzero(fde_change(fde), fde_change_size); fdunused(fdp, fd); +#ifdef CAPABILITIES + seq_write_end(&fde->fde_seq); +#endif } static inline void @@ -902,13 +909,19 @@ do_dup(struct thread *td, int flags, int old, int new, /* * Duplicate the source descriptor. */ +#ifdef CAPABILITIES + seq_write_begin(&newfde->fde_seq); +#endif filecaps_free(&newfde->fde_caps); - *newfde = *oldfde; + memcpy(fde_change(newfde), fde_change(oldfde), fde_change_size); filecaps_copy(&oldfde->fde_caps, &newfde->fde_caps); if ((flags & DUP_CLOEXEC) != 0) newfde->fde_flags = oldfde->fde_flags | UF_EXCLOSE; else newfde->fde_flags = oldfde->fde_flags & ~UF_EXCLOSE; +#ifdef CAPABILITIES + seq_write_end(&newfde->fde_seq); +#endif *retval = new; if (delfp != NULL) { @@ -1776,6 +1789,9 @@ finstall(struct thread *td, struct file *fp, int *fd, int flags, } fhold(fp); fde = &fdp->fd_ofiles[*fd]; +#ifdef CAPABILITIES + seq_write_begin(&fde->fde_seq); +#endif fde->fde_file = fp; if ((flags & O_CLOEXEC) != 0) fde->fde_flags |= UF_EXCLOSE; @@ -1783,6 +1799,9 @@ finstall(struct thread *td, struct file *fp, int *fd, int flags, filecaps_move(fcaps, &fde->fde_caps); else filecaps_fill(&fde->fde_caps); +#ifdef CAPABILITIES + seq_write_end(&fde->fde_seq); +#endif FILEDESC_XUNLOCK(fdp); return (0); } @@ -2315,6 +2334,7 @@ fget_unlocked(struct filedesc *fdp, int fd, cap_rights_t *needrightsp, struct file *fp; u_int count; #ifdef CAPABILITIES + seq_t seq; cap_rights_t haverights; int error; #endif @@ -2338,7 +2358,12 @@ fget_unlocked(struct filedesc *fdp, int fd, cap_rights_t *needrightsp, */ for (;;) { #ifdef CAPABILITIES + seq = seq_read(fd_seq(fdp, fd)); fde = fdt->fdt_ofiles[fd]; + if (!seq_consistent(fd_seq(fdp, fd), seq)) { + cpu_spinwait(); + continue; + } fp = fde.fde_file; #else fp = fdt->fdt_ofiles[fd].fde_file; @@ -2724,6 +2749,7 @@ int dupfdopen(struct thread *td, struct filedesc *fdp, int dfd, int mode, int openerror, int *indxp) { + struct filedescent *newfde, *oldfde; struct file *fp; int error, indx; @@ -2767,17 +2793,32 @@ dupfdopen(struct thread *td, struct filedesc *fdp, int dfd, int mode, return (EACCES); } fhold(fp); - fdp->fd_ofiles[indx] = fdp->fd_ofiles[dfd]; - filecaps_copy(&fdp->fd_ofiles[dfd].fde_caps, - &fdp->fd_ofiles[indx].fde_caps); + newfde = &fdp->fd_ofiles[indx]; + oldfde = &fdp->fd_ofiles[dfd]; +#ifdef CAPABILITIES + seq_write_begin(&newfde->fde_seq); +#endif + memcpy(fde_change(newfde), fde_change(oldfde), fde_change_size); + filecaps_copy(&oldfde->fde_caps, &newfde->fde_caps); +#ifdef CAPABILITIES + seq_write_end(&newfde->fde_seq); +#endif break; case ENXIO: /* * Steal away the file pointer from dfd and stuff it into indx. */ - fdp->fd_ofiles[indx] = fdp->fd_ofiles[dfd]; - bzero(&fdp->fd_ofiles[dfd], sizeof(fdp->fd_ofiles[dfd])); + newfde = &fdp->fd_ofiles[indx]; + oldfde = &fdp->fd_ofiles[dfd]; +#ifdef CAPABILITIES + seq_write_begin(&newfde->fde_seq); +#endif + memcpy(fde_change(newfde), fde_change(oldfde), fde_change_size); + bzero(fde_change(oldfde), fde_change_size); fdunused(fdp, dfd); +#ifdef CAPABILITIES + seq_write_end(&newfde->fde_seq); +#endif break; } FILEDESC_XUNLOCK(fdp); diff --git a/sys/sys/filedesc.h b/sys/sys/filedesc.h index 29cb6ae..a7f7737 100644 --- a/sys/sys/filedesc.h +++ b/sys/sys/filedesc.h @@ -33,11 +33,14 @@ #ifndef _SYS_FILEDESC_H_ #define _SYS_FILEDESC_H_ +#include "opt_capsicum.h" + #include #include #include #include #include +#include #include #include @@ -50,6 +53,9 @@ struct filecaps { }; struct filedescent { +#ifdef CAPABILITIES + seq_t fde_seq; /* if you need fde_file and fde_caps in sync */ +#endif struct file *fde_file; /* file structure for open file */ struct filecaps fde_caps; /* per-descriptor rights */ uint8_t fde_flags; /* per-process open file flags */ @@ -58,6 +64,13 @@ struct filedescent { #define fde_fcntls fde_caps.fc_fcntls #define fde_ioctls fde_caps.fc_ioctls #define fde_nioctls fde_caps.fc_nioctls +#ifdef CAPABILITIES +#define fde_change(fde) ((char *)(fde) + sizeof(seq_t)) +#define fde_change_size (sizeof(struct filedescent) - sizeof(seq_t)) +#else +#define fde_change(fde) ((fde)) +#define fde_change_size (sizeof(struct filedescent)) +#endif struct fdescenttbl { int fdt_nfiles; /* number of open files allocated */ @@ -88,6 +101,9 @@ struct filedesc { }; #define fd_nfiles fd_files->fdt_nfiles #define fd_ofiles fd_files->fdt_ofiles +#ifdef CAPABILITIES +#define fd_seq(fdp, fd) (&(fdp)->fd_ofiles[(fd)].fde_seq) +#endif /* * Structure to keep track of (process leader, struct fildedesc) tuples. -- 2.0.2 From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:55:17 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5448791; Fri, 15 Aug 2014 00:55:17 +0000 (UTC) Received: from mail-we0-x230.google.com (mail-we0-x230.google.com [IPv6:2a00:1450:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78E2B290D; Fri, 15 Aug 2014 00:55:16 +0000 (UTC) Received: by mail-we0-f176.google.com with SMTP id q58so1797467wes.21 for ; Thu, 14 Aug 2014 17:55:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=mA31eHG9q8vFE9MAdjYGo+jJEjzfRpzwJD87bwDsJv4=; b=GGNzXMRdKPUt8tbmLO7YhdxPxDzjsZ49d092bXcA1mplrZ0IkR6Dl+HmUwT1gs22y1 F1NiL9Tog+ZxYLrf0MI0M+f71df42bdKOLRvpFQ1FgU2CXLpGSpZjanmzrFACd+VEMgo yFNXouwXI/4UlGHUDinpHt3hRLT0vQ/DiHcUnHqPz/GCX41QPLWSuCsGt0FfdEarqPwK 8azCbT9EQi/ovhmHXUHlOXdVD/BSDlI+EzyoYUx6jlAKdAwfKe/l3KFdcnW0frnkdE2W ZMOaWooELo/mCSVRVU2e3sCyLNzLK/1Ebg35hJwPO0YIkxmDT+09dZ35xGkKUuc7ju43 f4rA== X-Received: by 10.194.6.101 with SMTP id z5mr17021092wjz.79.1408064114790; Thu, 14 Aug 2014 17:55:14 -0700 (PDT) Received: from localhost.localdomain (ip-62-245-66-51.net.upcbroadband.cz. [62.245.66.51]) by mx.google.com with ESMTPSA id fp6sm1473669wic.11.2014.08.14.17.55.13 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Aug 2014 17:55:13 -0700 (PDT) From: Mateusz Guzik To: freebsd-arch@freebsd.org Subject: [PATCH 0/2] plug capability races Date: Fri, 15 Aug 2014 02:55:10 +0200 Message-Id: <1408064112-573-1-git-send-email-mjguzik@gmail.com> X-Mailer: git-send-email 1.8.3.1 Cc: Robert Watson , Johan Schuijt , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:55:17 -0000 fget_unlocked currently reads 'fde' which is a structure consisting of serveral fields. In effect the read is inatomic and may result in obtaining file pointer with stale or incorrect capabilities. Example race is with dup2. Side effect is that capability checks can be circumvented. Proposed way to fix it is with the help of sequence counters. Patchset assumes stuff from 'Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked' ( http://lists.freebsd.org/pipermail/freebsd-arch/2014-July/015550.html ) is applied. There is no technical dependency between patches (apart from READ_ONCE), but this patch amortizes performance hit introduced with seqlock. So this introduces a measurable hit with a microbenchmark (16 threads reading from a pipe which fails with EAGAIN), but is still much faster than current code with atomic_load_acq_int(&fdp->fd_nfiles). x propernoacq-readpipe-run-sum + seq2-noacq-readpipe-run-sum N Min Max Median Avg Stddev x 20 59479718 59527286 59496714 59499504 13752.968 + 20 54520752 54920054 54829539 54773480 136842.96 Difference at 95.0% confidence -4.72602e+06 +/- 62244.4 -7.94296% +/- 0.104613% (Student's t, pooled s = 97250) There is still one theoretical race unfixed, but I don't believe it matters much. The race is: fp gets reallocated before refcount check. this resuls in returning fp regardless of new caps, but I don't see how this particular race could be exploited. It could be fixed by re-reading entire fde and checking if it changed. -- 2.0.2 From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 02:15:26 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB56FC3C; Fri, 15 Aug 2014 02:15:26 +0000 (UTC) Received: from mail108.syd.optusnet.com.au (mail108.syd.optusnet.com.au [211.29.132.59]) by mx1.freebsd.org (Postfix) with ESMTP id 4BD9F20EB; Fri, 15 Aug 2014 02:15:25 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail108.syd.optusnet.com.au (Postfix) with ESMTPS id DC3381A2044; Fri, 15 Aug 2014 12:15:16 +1000 (EST) Date: Fri, 15 Aug 2014 12:15:16 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Adrian Chadd Subject: Re: RFC: cpuid_t In-Reply-To: Message-ID: <20140815113509.F1151@besplex.bde.org> References: <201408120939.56176.jhb@freebsd.org> <201408141143.01137.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=BdjhjNd2 c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=6I5d2MoRAAAA:8 a=PdWYHDfVZEZrx_OGp8wA:9 a=Y3Ydrqqc73HhMB8v:21 a=i40qTmOqVJtlgkuw:21 a=CjuIK1q_8ugA:10 a=SV7veod9ZcQA:10 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 02:15:26 -0000 On Thu, 14 Aug 2014, Adrian Chadd wrote: > On 14 August 2014 08:43, John Baldwin wrote: >> On Tuesday, August 12, 2014 6:17:27 pm Adrian Chadd wrote: >>> On 12 August 2014 06:39, John Baldwin wrote: >>> >>>> I think Bruce's point is you can just fix the few places that use a u_char to >>>> an int and call it a day. The biggest offender of those that I can think of >>>> are td_oncpu/td_lastcpu and the corresponding fields in kinfo along with >>>> expanding NOCPU to be -1 instead of 255. I think most other places already >>>> use an int (pf is one place that doesn't, it borrows N bits from some other >>>> field to store the CPU number, so it can't use a cpuid_t anyway), so the patch >>>> to just use an int might actually be quite small. >>> >>> You could do that, but then we'd be left in a situation that it isn't >>> 100% clear what type people should be using (except hoping they >>> cut/paste code that uses an int and don't make dumb decisions like >>> assuming they can pack it into other fields, like pf) and that they >>> could treat the CPU ID as some kind of index number. Being able to >>> grep for all the places where cpuid_t is will make it a lot easier in >>> the future to uplift all of the cpu id manipulating code to treat CPU >>> IDs differently (eg making things very sparse.) >> >> I'm not sure how a typedef makes it easier to handle sparse IDs over an int? >> Plus, the IDs are already allowed to be sparse (hence CPU_ABSENT() and >> CPU_FOREACH(), etc.). I doubt that a typedef would make it any easier to >> deal with issues like pf either because the current code is not explicitly >> using any type at all, it's just passed into magic macros. > > It doesn't. It's there to help you find places where people aren't > doing the iteration correctly. > > You can do tricks like making cpuid_t temporarily a struct with the > int embedded in it: it'll work fine where it's being copied around as > an opaque type, but it'll compiler error out the minute some code > tries passing it into a field that takes an int, or a char. That is good for debugging and converting, but too complex to commit and maintain forever. It is only feasible because CPU ids are so rarely used. You wouldn't be able to do this for file descriptors. >>> So, I'm happy if the group decision is to just make it an int. It just >>> has to be done before FreeBSD-11 ships. :) >> >> I think making it an int would be a good first step at the very least. It >> would be good to see how many places don't already use it that way. >> >>> I don't have such an aversion to typedefs. It has always made it much >>> easier to do type checking and changes to things in C that I'd >>> normally do in C++ with classes. They aren't much use in C. >> One downside is that for printf() (a non-trivial thing in C), you have to >> either violate the abstraction (i.e. "know" it's an int) or use intmax_t casts >> everywhere. > > That's why you have a cpuid_t_to_quad() macro and use that everywhere > you need a numerical representation of the ID. That way it's up to the > macro or inline to return it as the type you require and thus printf() > and friends can have a well-known type in the string. That's mainly just harder to write and debug. > (I dislike intmax_t casts. I'd rather there were explicit ways to > convert those types to integer representations where appropriate > rather than having the code do those casts just to print. Ugh.) So you want everything to do explicit conversions and be more painful to write than a cast?: foo[cpuid_to_index(cpuid)] = ... printf("%qd", cpuid_to_quad(cpuid)); I made this more painful than necessary by providing more than 1 conversion function. Otherwise, the large quad type used to give a large fixed format for printing would unnecessarily pessimize everything. Conversion for indexing is only needed if cpuid is only a cookie that needs conversion. Bruce From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 00:41:15 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF6D5D8E; Fri, 15 Aug 2014 00:41:14 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D6D2B27B1; Fri, 15 Aug 2014 00:41:14 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 92753346DE3D; Thu, 14 Aug 2014 17:41:14 -0700 (PDT) Message-ID: <53ED578B.6070205@freebsd.org> Date: Thu, 14 Aug 2014 17:42:51 -0700 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Warner Losh , John Baldwin Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> <201408140847.00573.jhb@freebsd.org> <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> In-Reply-To: <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 15 Aug 2014 02:55:11 +0000 Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 00:41:15 -0000 On 8/14/14 9:13 AM, Warner Losh wrote: > On Aug 14, 2014, at 6:47 AM, John Baldwin wrote: > >>>> Marking the binary with a libxo-specific note tells the caller that >>>> the binary is capable of rendering its output in a non-traditional >>>> style and gives the caller a means of triggering those styles of >>>> output. In the libxo-enabled world, I see this as vital information >>>> the caller needs to initialize the environment in which the command >>>> will be run. Isn't this exactly the sort of information ELF targets >>>> for note sections? >>> How binary format has any relevance for an application level feature ? >>> What would you do with the binaries which permissions are 'r-s--x--x', >>> which is not unexpected for the tools which gather system information >>> and have to access things like /dev/mem ? >>> >>> You removed and did not answered a crusial question, which is a litmus >>> test for your proposal. Namely, how presence of the proposed note in >>> the binary is different from DT_NEEDED tag for your library ? >> Yes, checking DT_NEEDED for libxo.so is the first thing I thought of as well. >> It is equivalent to 'ldd foo | grep libxo'. > Doesn’t work for static binaries, nor for cases where libxo is linked in by a > library indirectly, nor for when the command is a shell script that may > invoke a command that supports this output, nor for a python script that > implements this output, etc. > > My question for people advocating this method: Why not require all commands > that generate this kind of output to support a standard command line option > that causes the command to print nothing and return 0 if it supports reporting, > or anything else if it doesn’t (return 0 with output, or return non-zero with or without > output). This would handle the more complicated implementation issues with using > DT_NEEDED and/or the ELF note, be more in line with how things are traditionally > done, and offer greater flexibility of implementation. > > Warner This is a decent idea, however the problem is firstly that most short-opts are taken, second issue is that adding getopt_long to a whole slew of programs will make the effort take a long time. It's really better to limit scope of this such that we are just making machine readable output. -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 05:36:17 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DAE01E59; Fri, 15 Aug 2014 05:36:17 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0190.outbound.protection.outlook.com [207.46.163.190]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D72882749; Fri, 15 Aug 2014 05:36:15 +0000 (UTC) Received: from BLUPR05CA010.namprd05.prod.outlook.com (10.255.219.168) by BLUPR05MB723.namprd05.prod.outlook.com (10.141.207.153) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Fri, 15 Aug 2014 05:36:07 +0000 Received: from BL2FFO11FD030.protection.gbl (2a01:111:f400:7c09::161) by BLUPR05CA010.outlook.office365.com (2a01:111:e400:83f::40) with Microsoft SMTP Server (TLS) id 15.0.1005.10 via Frontend Transport; Fri, 15 Aug 2014 05:36:07 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BL2FFO11FD030.mail.protection.outlook.com (10.173.161.40) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Fri, 15 Aug 2014 05:36:07 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 14 Aug 2014 22:36:06 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7F5a4n61377; Thu, 14 Aug 2014 22:36:04 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 9E40B580A2; Thu, 14 Aug 2014 22:36:04 -0700 (PDT) To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <53ED57F2.5020808@mu.org> References: <201408141640.s7EGe422096656@idle.juniper.net> <53ED57F2.5020808@mu.org> Comments: In-reply-to: Alfred Perlstein message dated "Thu, 14 Aug 2014 17:44:34 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 14 Aug 2014 22:36:04 -0700 Message-ID: <20140815053604.9E40B580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199003)(50226001)(90896003)(89996001)(92726001)(101356003)(81342001)(92566001)(69596002)(86362001)(87286001)(85306004)(20776003)(4396001)(76506005)(64706001)(77982001)(68736004)(57986006)(79102001)(84676001)(21056001)(80022001)(70486001)(44976005)(6806004)(83322001)(76482001)(74662001)(47776003)(31966008)(74502001)(87936001)(88136002)(46102001)(85852003)(95666004)(97736001)(76176999)(50986999)(105596002)(102836001)(48376002)(33656002)(93546004)(102176002)(81542001)(99396002)(104166001)(107046002)(93916002)(83072002)(77156001)(110136001)(81156004)(62966002)(106466001)(50466002)(42262002); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB723; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 0304E36CA3 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 05:36:18 -0000 On Thu, 14 Aug 2014 17:44:34 -0700, Alfred Perlstein writes: >I don't understand the need to query these programs for support of the >option. It's very simlpe. You run app A and tell it to output XML. App A needs to run app B, and needs to know if it can output valid XML or whether all its output needs to be wrapped/escaped. Failure to handle this will result in garbage. From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 07:04:00 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0DDDF88; Fri, 15 Aug 2014 07:03:59 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id DA0942065; Fri, 15 Aug 2014 07:03:59 +0000 (UTC) Received: from AlfredMacbookAir.local (c-76-21-10-192.hsd1.ca.comcast.net [76.21.10.192]) by elvis.mu.org (Postfix) with ESMTPSA id DE129346DE73; Fri, 15 Aug 2014 00:03:58 -0700 (PDT) Message-ID: <53EDB0EF.6090902@mu.org> Date: Fri, 15 Aug 2014 00:04:15 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408141640.s7EGe422096656@idle.juniper.net> <53ED57F2.5020808@mu.org> <20140815053604.9E40B580A2@chaos.jnpr.net> In-Reply-To: <20140815053604.9E40B580A2@chaos.jnpr.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 07:04:00 -0000 On 8/14/14, 10:36 PM, Simon J. Gerraty wrote: > On Thu, 14 Aug 2014 17:44:34 -0700, Alfred Perlstein writes: >> I don't understand the need to query these programs for support of the >> option. > It's very simlpe. You run app A and tell it to output XML. > App A needs to run app B, and needs to know if it can output valid XML > or whether all its output needs to be wrapped/escaped. > Failure to handle this will result in garbage. > > Sure that is fairly simple, but where will that really even be needed in practice right now? It seems fairly contrived, for what use case? Something like "find / --type libxo --exec {} \;" ? Or something real? This really seems to be going off the deep end of over engineering. How many programs have been successfully converted over to libxo at this point? -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 08:25:14 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 833F09C3 for ; Fri, 15 Aug 2014 08:25:14 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06E4128F5 for ; Fri, 15 Aug 2014 08:25:13 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7F8P359003609 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 15 Aug 2014 11:25:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7F8P359003609 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7F8P1Qe003595; Fri, 15 Aug 2014 11:25:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Aug 2014 11:25:01 +0300 From: Konstantin Belousov To: Marcel Moolenaar Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140815082501.GR2737@kib.kiev.ua> References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6nfUOKxk8V1TQsEu" Content-Disposition: inline In-Reply-To: <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 08:25:14 -0000 --6nfUOKxk8V1TQsEu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 14, 2014 at 10:04:36AM -0700, Marcel Moolenaar wrote: >=20 > On Aug 13, 2014, at 10:26 PM, Konstantin Belousov w= rote: > >>=20 > >> I know ELF "Note" elements can be used to carry vendor-specific > >> data, but have no experience with them. Would it be reasonable to > >> use them as a means of communicating this information to other bits > >> of software? > > No. >=20 > Too extreme. >=20 > >> Is FreeBSD using Notes for other information currently? > > Yes, the notes are used to communicate the information required by > > the dynamic linker to correctly activate the image. The mechanism has > > nothing to do with application-specific features, and overloading it for > > that purpose is severe and pointless layering violation. Things should > > not be done just because they could be done. >=20 > Too extreme. Life is a lot more subtle. Standards > are as well. There are many examples in the real > world where standards are interpreted a little > more liberal than others may want to. When such > result in (gratuitous) incompatibilities, we all > interpret it as bad. But when it adds real value, > you tend to find it in the next update of the > standard. Do you suggest that next revision of ELF standard starts talking about app-level features ? >=20 > > Using the static tagging for the dynamic application properties is wrong > > anyway. E.g., would you consider the mere fact that the binary is link= ed > > against your library, as the indication that your feature is supported ? > > If not, how does it differ from the presence of some additional note ? >=20 > If we can eliminate static linking for libxo, than > that is definitely easy. Easiest probably. The > question becomes: is it acceptable to not support > static linking for libxo? Or alternatively, is it > acceptable to not be able to check for the feature > on a static executable? Let me clarify. I do not suggest to use DT_NEEDED for libxo (?) as the alternative feature test, either. I consider both the proposed note and the presence of dependency on libxo.so as equally wrong ways to determine the perceived support for specific output formats. My point was that note is essentially same as DT_NEEDED tag, with the same set of architectural problems, but it is easier to see them for DT_NEEDED. Static binaries is just a detail, and yes, I consider static linking as place where we cannot hold an ABI guarantee. The good example was kse libpthread, which broke statically linked threaded binaries. My objections against use of ELF could be summarized as following: - ELF is a binary standard for image activation and runtime system, not for the app level features. - Attempt to shoe-horn the discussed feature into binary format makes the indicator cut of the actual feature. It is like the mark on the plastic can, which may have no relation to the can content. From this PoV, the options or special env variables are at the right place. - It is unportable. Why bind the pure data transformation library to the platform ? What about Mach-O or COFF/PE platforms ? What if FreeBSD decide to change binary format, or add support for a new format, besides ELF ? What if the tool and consumer are of different ABI ? E.g., one is 64 bit, another is 32bit. Usually, there is support for less bits, but 32bit libelf or debugger or libunwind or whatever usually have troubles reading 64bit ELF objects. >=20 > For the first I'm inclined to say yes, but not for > the second. In fact, since I already started talking on the subject, I dislike the idea of adding the other text formats to existing tools. Much more reasonable route, IMO, is to have a library to interrogate system state and present the data in ABI-stable and introspection-friendly way, without enforcing the kernel->internal userspace structure->text serialization-> text parsing ->some other internal userspace structure path on the consumers. If so inclined, some existing tools could be converted to use the library and possibly libxo. But I suspect that the presence of the library for system state and bindings for the most popular scripting languages (which is facilitated by the introspection facilities, to avoid the need of translating each ABI structure into each language datatype) is the sweet spot for the current consumers of libxo formatted output. There would be no need for running external tool, determining if it is suitable, parsing its output, treat the text output as ABI contract and so on. --6nfUOKxk8V1TQsEu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT7cPdAAoJEJDCuSvBvK1B4foP/jKJseCkwkXr1JZGiVdv/wjW 5jAeJCZHuBo7sqiFXAIWXZbbbJHcX9ww30HiDBjWLPAk/tsGguzgNiK7EeFXeKj1 bipABPMRWvspBJ4HgvfwFcNJ0iNq24EaMobh1tG2oby4OhIaS/ZbXxR3L0kbkAYl PMmEemWSuJZM+K0DxYuonVXm+YKrU+6GUZgDC6Sm/FvJUDhTu4wir0xNLEEz1ioi 5sSe1oKByxlItVKuvzIhtRX2j3tHhWcn+e13EC4fuEL0pufHyQUWyEfpUFmEuxxy lqBVlTTeubBhpL7FT6MvBu0JEu673q4ofsmDy76J2rBbEXG3BuslisZk7u38gg90 0lNuZekRZAFvjFtDTXaqYztkUoGNvZI4IR31fK835tjZVwtImxvLCtOYIJQ/1MDK KMTLIniglDnF1xbjngvj4kkShPe1YEAxPkO2RkD7LjZZp6ksIballGdjWFKeyImX PCenSmsK1j0YqCe5YWAuKbutoVQT2IT93m6RbFHibzjXf8RjJf5X67ZFitRSodFN 4bZM5WK/mrwejPlDJBIxcReOIpu9AkyOuOCZwnVJf3mR0AXoW9MzKJkL1npipbim wFHhj1Sy5pdgCU0Xr3mGW9gdvV0e/DJd4ebLcPa8vWvfxbQOvO1VoU7uU190wmA5 fL00mWEYBlQWNDHyQJs7 =RDy9 -----END PGP SIGNATURE----- --6nfUOKxk8V1TQsEu-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 13:20:31 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A3847DD for ; Fri, 15 Aug 2014 13:20:31 +0000 (UTC) Received: from mail-qa0-x22a.google.com (mail-qa0-x22a.google.com [IPv6:2607:f8b0:400d:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC6A62C7F for ; Fri, 15 Aug 2014 13:20:30 +0000 (UTC) Received: by mail-qa0-f42.google.com with SMTP id j15so2059263qaq.15 for ; Fri, 15 Aug 2014 06:20:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=HDubMFhgV7xoFe4W6wqacq52TEDyi+XPggZG87QbuOk=; b=G5/E6ceQBVoYFiVLPO9X109jfWVlv1vr7wZnSuw5UYflTa6FJuJkcWNLSAvg0RuaDF 5y6g6Ola2nBUb3+RvVl8DptrIMiPr5i+hn6sGF+yLtqbHcXGPKJnxV2p5I1e28UedqNs vxMI4XhFi2iGOUMoaHggcTr84q6Lu8Uo2m68U8zlDHFvYvaEr5Oocx4AU+i6ATEfbldX em7jjrl7tmNIlJNnq5OfK1mrpzzCrAfOLHMtcJyRifwNqBGouH7y3i6YrmqeYqPCoJRu SSGlILfiki2+HQiwzCC9cisWEeFcvVGD67MeK38a1I+YN2hEIXqjw7k/Nac9QhjztgXo nKAg== MIME-Version: 1.0 X-Received: by 10.140.29.138 with SMTP id b10mr26230129qgb.15.1408108829795; Fri, 15 Aug 2014 06:20:29 -0700 (PDT) Received: by 10.140.33.137 with HTTP; Fri, 15 Aug 2014 06:20:29 -0700 (PDT) Date: Fri, 15 Aug 2014 15:20:29 +0200 Message-ID: Subject: PMAP_LOCK_DESTROY() vs. vmspace UMA_ZONE_NOFREE From: Svatopluk Kraus To: freebsd-arch@freebsd.org Content-Type: multipart/mixed; boundary=001a113a75a29b3e060500aae22f X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 13:20:31 -0000 --001a113a75a29b3e060500aae22f Content-Type: text/plain; charset=UTF-8 Hi, I was playing with ports/benchmarks/forkbomb while testing my and Michal's armv6 pmap based on i386 one when I found - IMHO - sketelot in the closet. i386, xen, and sparc64 pmaps call PMAP_LOCK_DESTROY() in pmap_pinit() if some allocation failed. As vmspace (struct pmap is part of it) uma zone is created with UMA_ZONE_NOFREE flag and PMAP_LOCK_INIT() is called from vmspace_zinit() initiator, PMAP_LOCK_DESTROY() should be called from nowhere. The pmap_c_patch.txt is attached to solve it. IMHO, I think that definition of PMAP_LOCK_DESTROY() is misleading a little as PMAP_LOCK_DESTROY() cannot be used anywhere as long as UMA_ZONE_NOFREE flag is used. The pmap_all_patch.txt is attach to wipe PMAP_LOCK_DESTROY() out of source tree. Svata --001a113a75a29b3e060500aae22f Content-Type: text/plain; charset=US-ASCII; name="pmap_c_patch.txt" Content-Disposition: attachment; filename="pmap_c_patch.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hyvjlr090 SW5kZXg6IHN5cy9pMzg2L2kzODYvcG1hcC5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5cy9pMzg2L2kzODYv cG1hcC5jCShyZXZpc2lvbiAyNzAwMjApCisrKyBzeXMvaTM4Ni9pMzg2L3BtYXAuYwkod29ya2lu ZyBjb3B5KQpAQCAtMTc1NSwxMCArMTc1NSw4IEBACiAJICovCiAJaWYgKHBtYXAtPnBtX3BkaXIg PT0gTlVMTCkgewogCQlwbWFwLT5wbV9wZGlyID0gKHBkX2VudHJ5X3QgKilrdmFfYWxsb2MoTkJQ VEQpOwotCQlpZiAocG1hcC0+cG1fcGRpciA9PSBOVUxMKSB7Ci0JCQlQTUFQX0xPQ0tfREVTVFJP WShwbWFwKTsKKwkJaWYgKHBtYXAtPnBtX3BkaXIgPT0gTlVMTCkKIAkJCXJldHVybiAoMCk7Ci0J CX0KICNpZmRlZiBQQUUKIAkJcG1hcC0+cG1fcGRwdCA9IHVtYV96YWxsb2MocGRwdHpvbmUsIE1f V0FJVE9LIHwgTV9aRVJPKTsKIAkJS0FTU0VSVCgoKHZtX29mZnNldF90KXBtYXAtPnBtX3BkcHQg JgpJbmRleDogc3lzL2kzODYveGVuL3BtYXAuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvaTM4Ni94ZW4v cG1hcC5jCShyZXZpc2lvbiAyNzAwMjApCisrKyBzeXMvaTM4Ni94ZW4vcG1hcC5jCSh3b3JraW5n IGNvcHkpCkBAIC0xNDU5LDcgKzE0NTksNiBAQAogCWlmIChwbWFwLT5wbV9wZGlyID09IE5VTEwp IHsKIAkJcG1hcC0+cG1fcGRpciA9IChwZF9lbnRyeV90ICopa3ZhX2FsbG9jKE5CUFREKTsKIAkJ aWYgKHBtYXAtPnBtX3BkaXIgPT0gTlVMTCkgewotCQkJUE1BUF9MT0NLX0RFU1RST1kocG1hcCk7 CiAjaWZkZWYgSEFNRklTVEVEX0xPQ0tJTkcKIAkJCW10eF91bmxvY2soJmNyZWF0ZWRlbGV0ZV9s b2NrKTsKICNlbmRpZgpJbmRleDogc3lzL3NwYXJjNjQvc3BhcmM2NC9wbWFwLmMKPT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PQotLS0gc3lzL3NwYXJjNjQvc3BhcmM2NC9wbWFwLmMJKHJldmlzaW9uIDI3MDAyMCkKKysrIHN5 cy9zcGFyYzY0L3NwYXJjNjQvcG1hcC5jCSh3b3JraW5nIGNvcHkpCkBAIC0xMjExLDExICsxMjEx LDkgQEAKIAkgKi8KIAlpZiAocG0tPnBtX3RzYiA9PSBOVUxMKSB7CiAJCXBtLT5wbV90c2IgPSAo c3RydWN0IHR0ZSAqKWt2YV9hbGxvYyhUU0JfQlNJWkUpOwotCQlpZiAocG0tPnBtX3RzYiA9PSBO VUxMKSB7Ci0JCQlQTUFQX0xPQ0tfREVTVFJPWShwbSk7CisJCWlmIChwbS0+cG1fdHNiID09IE5V TEwpCiAJCQlyZXR1cm4gKDApOwogCQl9Ci0JfQogCiAJLyoKIAkgKiBBbGxvY2F0ZSBhbiBvYmpl Y3QgZm9yIGl0Lgo= --001a113a75a29b3e060500aae22f Content-Type: text/plain; charset=US-ASCII; name="pmap_all_patch.txt" Content-Disposition: attachment; filename="pmap_all_patch.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hyvjlvdd1 SW5kZXg6IHN5cy9hbWQ2NC9pbmNsdWRlL3BtYXAuaAo9PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvYW1kNjQv aW5jbHVkZS9wbWFwLmgJKHJldmlzaW9uIDI3MDAyMCkKKysrIHN5cy9hbWQ2NC9pbmNsdWRlL3Bt YXAuaAkod29ya2luZyBjb3B5KQpAQCAtMzI2LDcgKzMyNiw2IEBACiAjZGVmaW5lCVBNQVBfTE9D SyhwbWFwKQkJbXR4X2xvY2soJihwbWFwKS0+cG1fbXR4KQogI2RlZmluZQlQTUFQX0xPQ0tfQVNT RVJUKHBtYXAsIHR5cGUpIFwKIAkJCQltdHhfYXNzZXJ0KCYocG1hcCktPnBtX210eCwgKHR5cGUp KQotI2RlZmluZQlQTUFQX0xPQ0tfREVTVFJPWShwbWFwKQltdHhfZGVzdHJveSgmKHBtYXApLT5w bV9tdHgpCiAjZGVmaW5lCVBNQVBfTE9DS19JTklUKHBtYXApCW10eF9pbml0KCYocG1hcCktPnBt X210eCwgInBtYXAiLCBcCiAJCQkJICAgIE5VTEwsIE1UWF9ERUYgfCBNVFhfRFVQT0spCiAjZGVm aW5lCVBNQVBfTE9DS0VEKHBtYXApCW10eF9vd25lZCgmKHBtYXApLT5wbV9tdHgpCkluZGV4OiBz eXMvYXJtL2luY2x1ZGUvcG1hcC5oCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5cy9hcm0vaW5jbHVkZS9wbWFw LmgJKHJldmlzaW9uIDI3MDAyMCkKKysrIHN5cy9hcm0vaW5jbHVkZS9wbWFwLmgJKHdvcmtpbmcg Y29weSkKQEAgLTE3Niw3ICsxNzYsNiBAQAogI2RlZmluZQlQTUFQX0FTU0VSVF9MT0NLRUQocG1h cCkgXAogCQkJCW10eF9hc3NlcnQoJihwbWFwKS0+cG1fbXR4LCBNQV9PV05FRCkKICNkZWZpbmUJ UE1BUF9MT0NLKHBtYXApCQltdHhfbG9jaygmKHBtYXApLT5wbV9tdHgpCi0jZGVmaW5lCVBNQVBf TE9DS19ERVNUUk9ZKHBtYXApCW10eF9kZXN0cm95KCYocG1hcCktPnBtX210eCkKICNkZWZpbmUJ UE1BUF9MT0NLX0lOSVQocG1hcCkJbXR4X2luaXQoJihwbWFwKS0+cG1fbXR4LCAicG1hcCIsIFwK IAkJCQkgICAgTlVMTCwgTVRYX0RFRiB8IE1UWF9EVVBPSykKICNkZWZpbmUJUE1BUF9PV05FRChw bWFwKQltdHhfb3duZWQoJihwbWFwKS0+cG1fbXR4KQpJbmRleDogc3lzL2kzODYvaTM4Ni9wbWFw LmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PQotLS0gc3lzL2kzODYvaTM4Ni9wbWFwLmMJKHJldmlzaW9uIDI3MDAyMCkK KysrIHN5cy9pMzg2L2kzODYvcG1hcC5jCSh3b3JraW5nIGNvcHkpCkBAIC0xNzU1LDEwICsxNzU1 LDggQEAKIAkgKi8KIAlpZiAocG1hcC0+cG1fcGRpciA9PSBOVUxMKSB7CiAJCXBtYXAtPnBtX3Bk aXIgPSAocGRfZW50cnlfdCAqKWt2YV9hbGxvYyhOQlBURCk7Ci0JCWlmIChwbWFwLT5wbV9wZGly ID09IE5VTEwpIHsKLQkJCVBNQVBfTE9DS19ERVNUUk9ZKHBtYXApOworCQlpZiAocG1hcC0+cG1f cGRpciA9PSBOVUxMKQogCQkJcmV0dXJuICgwKTsKLQkJfQogI2lmZGVmIFBBRQogCQlwbWFwLT5w bV9wZHB0ID0gdW1hX3phbGxvYyhwZHB0em9uZSwgTV9XQUlUT0sgfCBNX1pFUk8pOwogCQlLQVNT RVJUKCgodm1fb2Zmc2V0X3QpcG1hcC0+cG1fcGRwdCAmCkluZGV4OiBzeXMvaTM4Ni9pbmNsdWRl L3BtYXAuaAo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvaTM4Ni9pbmNsdWRlL3BtYXAuaAkocmV2aXNpb24g MjcwMDIwKQorKysgc3lzL2kzODYvaW5jbHVkZS9wbWFwLmgJKHdvcmtpbmcgY29weSkKQEAgLTM4 Niw3ICszODYsNiBAQAogI2RlZmluZQlQTUFQX0xPQ0socG1hcCkJCW10eF9sb2NrKCYocG1hcCkt PnBtX210eCkKICNkZWZpbmUJUE1BUF9MT0NLX0FTU0VSVChwbWFwLCB0eXBlKSBcCiAJCQkJbXR4 X2Fzc2VydCgmKHBtYXApLT5wbV9tdHgsICh0eXBlKSkKLSNkZWZpbmUJUE1BUF9MT0NLX0RFU1RS T1kocG1hcCkJbXR4X2Rlc3Ryb3koJihwbWFwKS0+cG1fbXR4KQogI2RlZmluZQlQTUFQX0xPQ0tf SU5JVChwbWFwKQltdHhfaW5pdCgmKHBtYXApLT5wbV9tdHgsICJwbWFwIiwgXAogCQkJCSAgICBO VUxMLCBNVFhfREVGIHwgTVRYX0RVUE9LKQogI2RlZmluZQlQTUFQX0xPQ0tFRChwbWFwKQltdHhf b3duZWQoJihwbWFwKS0+cG1fbXR4KQpJbmRleDogc3lzL2kzODYveGVuL3BtYXAuYwo9PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09Ci0tLSBzeXMvaTM4Ni94ZW4vcG1hcC5jCShyZXZpc2lvbiAyNzAwMjApCisrKyBzeXMvaTM4 Ni94ZW4vcG1hcC5jCSh3b3JraW5nIGNvcHkpCkBAIC0xNDU5LDcgKzE0NTksNiBAQAogCWlmIChw bWFwLT5wbV9wZGlyID09IE5VTEwpIHsKIAkJcG1hcC0+cG1fcGRpciA9IChwZF9lbnRyeV90ICop a3ZhX2FsbG9jKE5CUFREKTsKIAkJaWYgKHBtYXAtPnBtX3BkaXIgPT0gTlVMTCkgewotCQkJUE1B UF9MT0NLX0RFU1RST1kocG1hcCk7CiAjaWZkZWYgSEFNRklTVEVEX0xPQ0tJTkcKIAkJCW10eF91 bmxvY2soJmNyZWF0ZWRlbGV0ZV9sb2NrKTsKICNlbmRpZgpJbmRleDogc3lzL21pcHMvaW5jbHVk ZS9wbWFwLmgKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL21pcHMvaW5jbHVkZS9wbWFwLmgJKHJldmlzaW9u IDI3MDAyMCkKKysrIHN5cy9taXBzL2luY2x1ZGUvcG1hcC5oCSh3b3JraW5nIGNvcHkpCkBAIC0x MDYsNyArMTA2LDYgQEAKIAogI2RlZmluZQlQTUFQX0xPQ0socG1hcCkJCW10eF9sb2NrKCYocG1h cCktPnBtX210eCkKICNkZWZpbmUJUE1BUF9MT0NLX0FTU0VSVChwbWFwLCB0eXBlKQltdHhfYXNz ZXJ0KCYocG1hcCktPnBtX210eCwgKHR5cGUpKQotI2RlZmluZQlQTUFQX0xPQ0tfREVTVFJPWShw bWFwKSBtdHhfZGVzdHJveSgmKHBtYXApLT5wbV9tdHgpCiAjZGVmaW5lCVBNQVBfTE9DS19JTklU KHBtYXApCW10eF9pbml0KCYocG1hcCktPnBtX210eCwgInBtYXAiLCBcCiAJCQkJICAgIE5VTEws IE1UWF9ERUYpCiAjZGVmaW5lCVBNQVBfTE9DS0VEKHBtYXApCW10eF9vd25lZCgmKHBtYXApLT5w bV9tdHgpCkluZGV4OiBzeXMvcG93ZXJwYy9pbmNsdWRlL3BtYXAuaAo9PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBz eXMvcG93ZXJwYy9pbmNsdWRlL3BtYXAuaAkocmV2aXNpb24gMjcwMDIwKQorKysgc3lzL3Bvd2Vy cGMvaW5jbHVkZS9wbWFwLmgJKHdvcmtpbmcgY29weSkKQEAgLTIxMyw3ICsyMTMsNiBAQAogI2Rl ZmluZQlQTUFQX0xPQ0socG1hcCkJCW10eF9sb2NrKCYocG1hcCktPnBtX210eCkKICNkZWZpbmUJ UE1BUF9MT0NLX0FTU0VSVChwbWFwLCB0eXBlKSBcCiAJCQkJbXR4X2Fzc2VydCgmKHBtYXApLT5w bV9tdHgsICh0eXBlKSkKLSNkZWZpbmUJUE1BUF9MT0NLX0RFU1RST1kocG1hcCkJbXR4X2Rlc3Ry b3koJihwbWFwKS0+cG1fbXR4KQogI2RlZmluZQlQTUFQX0xPQ0tfSU5JVChwbWFwKQltdHhfaW5p dCgmKHBtYXApLT5wbV9tdHgsIFwKIAkJCQkgICAgKHBtYXAgPT0ga2VybmVsX3BtYXApID8gImtl cm5lbHBtYXAiIDogXAogCQkJCSAgICAicG1hcCIsIE5VTEwsIE1UWF9ERUYpCkluZGV4OiBzeXMv c3BhcmM2NC9pbmNsdWRlL3BtYXAuaAo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvc3BhcmM2NC9pbmNsdWRl L3BtYXAuaAkocmV2aXNpb24gMjcwMDIwKQorKysgc3lzL3NwYXJjNjQvaW5jbHVkZS9wbWFwLmgJ KHdvcmtpbmcgY29weSkKQEAgLTcwLDcgKzcwLDYgQEAKICNkZWZpbmUJUE1BUF9MT0NLKHBtYXAp CQltdHhfbG9jaygmKHBtYXApLT5wbV9tdHgpCiAjZGVmaW5lCVBNQVBfTE9DS19BU1NFUlQocG1h cCwgdHlwZSkJCQkJCVwKIAkJCQltdHhfYXNzZXJ0KCYocG1hcCktPnBtX210eCwgKHR5cGUpKQot I2RlZmluZQlQTUFQX0xPQ0tfREVTVFJPWShwbWFwKQltdHhfZGVzdHJveSgmKHBtYXApLT5wbV9t dHgpCiAjZGVmaW5lCVBNQVBfTE9DS19JTklUKHBtYXApCW10eF9pbml0KCYocG1hcCktPnBtX210 eCwgInBtYXAiLAlcCiAJCQkJICAgIE5VTEwsIE1UWF9ERUYgfCBNVFhfRFVQT0spCiAjZGVmaW5l CVBNQVBfTE9DS0VEKHBtYXApCW10eF9vd25lZCgmKHBtYXApLT5wbV9tdHgpCkluZGV4OiBzeXMv c3BhcmM2NC9zcGFyYzY0L3BtYXAuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvc3BhcmM2NC9zcGFyYzY0 L3BtYXAuYwkocmV2aXNpb24gMjcwMDIwKQorKysgc3lzL3NwYXJjNjQvc3BhcmM2NC9wbWFwLmMJ KHdvcmtpbmcgY29weSkKQEAgLTEyMTEsMTEgKzEyMTEsOSBAQAogCSAqLwogCWlmIChwbS0+cG1f dHNiID09IE5VTEwpIHsKIAkJcG0tPnBtX3RzYiA9IChzdHJ1Y3QgdHRlICopa3ZhX2FsbG9jKFRT Ql9CU0laRSk7Ci0JCWlmIChwbS0+cG1fdHNiID09IE5VTEwpIHsKLQkJCVBNQVBfTE9DS19ERVNU Uk9ZKHBtKTsKKwkJaWYgKHBtLT5wbV90c2IgPT0gTlVMTCkKIAkJCXJldHVybiAoMCk7CiAJCX0K LQl9CiAKIAkvKgogCSAqIEFsbG9jYXRlIGFuIG9iamVjdCBmb3IgaXQuCg== --001a113a75a29b3e060500aae22f-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 14:02:30 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D6C644B for ; Fri, 15 Aug 2014 14:02:30 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98BF0221F for ; Fri, 15 Aug 2014 14:02:29 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7FE2Jeq011615 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 15 Aug 2014 17:02:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7FE2Jeq011615 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7FE2J0u011614; Fri, 15 Aug 2014 17:02:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 15 Aug 2014 17:02:19 +0300 From: Konstantin Belousov To: Svatopluk Kraus Subject: Re: PMAP_LOCK_DESTROY() vs. vmspace UMA_ZONE_NOFREE Message-ID: <20140815140219.GU2737@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KJvkvZqQCzHgjKcr" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 14:02:30 -0000 --KJvkvZqQCzHgjKcr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 15, 2014 at 03:20:29PM +0200, Svatopluk Kraus wrote: > Hi, >=20 > I was playing with ports/benchmarks/forkbomb while testing my and Michal's > armv6 pmap based on i386 one when I found - IMHO - sketelot in the closet. >=20 > i386, xen, and sparc64 pmaps call PMAP_LOCK_DESTROY() in pmap_pinit() if > some allocation failed. As vmspace (struct pmap is part of it) uma zone is > created with UMA_ZONE_NOFREE flag and PMAP_LOCK_INIT() is called from > vmspace_zinit() initiator, PMAP_LOCK_DESTROY() should be called from > nowhere. >=20 > The pmap_c_patch.txt is attached to solve it. >=20 > IMHO, I think that definition of PMAP_LOCK_DESTROY() is misleading a litt= le > as PMAP_LOCK_DESTROY() cannot be used anywhere as long as UMA_ZONE_NOFREE > flag is used. >=20 > The pmap_all_patch.txt is attach to wipe PMAP_LOCK_DESTROY() out of source > tree. >=20 > Svata > Index: sys/i386/i386/pmap.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/i386/i386/pmap.c (revision 270020) > +++ sys/i386/i386/pmap.c (working copy) > @@ -1755,10 +1755,8 @@ > */ > if (pmap->pm_pdir =3D=3D NULL) { > pmap->pm_pdir =3D (pd_entry_t *)kva_alloc(NBPTD); > - if (pmap->pm_pdir =3D=3D NULL) { > - PMAP_LOCK_DESTROY(pmap); > + if (pmap->pm_pdir =3D=3D NULL) > return (0); > - } > #ifdef PAE > pmap->pm_pdpt =3D uma_zalloc(pdptzone, M_WAITOK | M_ZERO); > KASSERT(((vm_offset_t)pmap->pm_pdpt & > Index: sys/i386/xen/pmap.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/i386/xen/pmap.c (revision 270020) > +++ sys/i386/xen/pmap.c (working copy) > @@ -1459,7 +1459,6 @@ > if (pmap->pm_pdir =3D=3D NULL) { > pmap->pm_pdir =3D (pd_entry_t *)kva_alloc(NBPTD); > if (pmap->pm_pdir =3D=3D NULL) { > - PMAP_LOCK_DESTROY(pmap); > #ifdef HAMFISTED_LOCKING > mtx_unlock(&createdelete_lock); > #endif > Index: sys/sparc64/sparc64/pmap.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/sparc64/sparc64/pmap.c (revision 270020) > +++ sys/sparc64/sparc64/pmap.c (working copy) > @@ -1211,11 +1211,9 @@ > */ > if (pm->pm_tsb =3D=3D NULL) { > pm->pm_tsb =3D (struct tte *)kva_alloc(TSB_BSIZE); > - if (pm->pm_tsb =3D=3D NULL) { > - PMAP_LOCK_DESTROY(pm); > + if (pm->pm_tsb =3D=3D NULL) > return (0); > } > - } > =20 > /* > * Allocate an object for it. Yes, I think this is result of incomplete r254667. Still, I prefer to keep the PMAP_LOCK_DESTROY() macro around. --KJvkvZqQCzHgjKcr Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT7hLrAAoJEJDCuSvBvK1BozcP/3BY1eiPVKmRjaLHJyOcV2Mt toCs/a3MtUMnCu1xTs4eV0T+L7cEZNoL14gqk6TuIzw1rf2o9+6Dfll1ppBCmiZ8 uYvrKmgXs506wLw5Ufi5AtB00HOES3C7KrDih6TJ61xqemfJNnKbpoSljLq4hrJj PYLzMhqFFQFTHS8zsQGE4XkV47UUy4sWqwpVWo9TyWxumAyy7faU3ukHlp7oJoUZ FbgZcDzr1Cpp8snZ6Udn0g8LuPqQaiZbV0dl57911MoRJMari+uRgOX98HtsgCxi 4Wmb+Bf2cUIdNd2zDS/UQB3NqRl3blqbnLO3NV52vicrBB8j36C+KGrQ5RSPWkht sIi+bnzq2MXZkFd2oQsUO8n4SUuNZ3ZzinP/6NwlEjACBgshkQU8Z+GbRvGr/HvE M9SF3HKx9oIN1bHKpO3sSUC53kx/uVgEjckmF8VzBGDOeX6+zwaRmIZb48sHfQDd 132ILaTKhoymQ+B3Nm2yhzwBPHRo0/Daj9RhV4HHQxxgd5DFbrMzhi4XwTp5lm7x nXwQfevxcMNZ8fbjtK0d6me1ULiaJF4CeBQrzO6SwqG70hF3+7LC4rV7YSBa78fp 5kQ0cuUJ1ot0lDu9RocbPzsVnx5MHAsysPkIh8A/IJnOSQm3I72yHhIzYg6dNRA/ 5ahRvbfprMjBcRnFHrc7 =L7Co -----END PGP SIGNATURE----- --KJvkvZqQCzHgjKcr-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 14:45:07 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00E2CDE4; Fri, 15 Aug 2014 14:45:06 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id DE813287D; Fri, 15 Aug 2014 14:45:06 +0000 (UTC) Received: from [10.0.1.107] (c-76-21-10-192.hsd1.ca.comcast.net [76.21.10.192]) by elvis.mu.org (Postfix) with ESMTPSA id EFC98346DE8B; Fri, 15 Aug 2014 07:45:05 -0700 (PDT) References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> <20140815082501.GR2737@kib.kiev.ua> Mime-Version: 1.0 (1.0) In-Reply-To: <20140815082501.GR2737@kib.kiev.ua> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPhone Mail (11D257) From: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Fri, 15 Aug 2014 07:44:52 -0700 To: Konstantin Belousov Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , "arch@freebsd.org" , Poul-Henning Kamp , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 14:45:07 -0000 Agree with kib.=20 Kib, one thing that I just don't understand is the point of even being able t= o query programs for libxo support. Does it make sense to you? =20 Sent from my iPhone > On Aug 15, 2014, at 1:25 AM, Konstantin Belousov wro= te: >=20 >> On Thu, Aug 14, 2014 at 10:04:36AM -0700, Marcel Moolenaar wrote: >>=20 >> On Aug 13, 2014, at 10:26 PM, Konstantin Belousov w= rote: >>>>=20 >>>> I know ELF "Note" elements can be used to carry vendor-specific >>>> data, but have no experience with them. Would it be reasonable to >>>> use them as a means of communicating this information to other bits >>>> of software? >>> No. >>=20 >> Too extreme. >>=20 >>>> Is FreeBSD using Notes for other information currently? >>> Yes, the notes are used to communicate the information required by >>> the dynamic linker to correctly activate the image. The mechanism has >>> nothing to do with application-specific features, and overloading it for= >>> that purpose is severe and pointless layering violation. Things should >>> not be done just because they could be done. >>=20 >> Too extreme. Life is a lot more subtle. Standards >> are as well. There are many examples in the real >> world where standards are interpreted a little >> more liberal than others may want to. When such >> result in (gratuitous) incompatibilities, we all >> interpret it as bad. But when it adds real value, >> you tend to find it in the next update of the >> standard. >=20 > Do you suggest that next revision of ELF standard starts talking > about app-level features ? >=20 >>=20 >>> Using the static tagging for the dynamic application properties is wrong= >>> anyway. E.g., would you consider the mere fact that the binary is linke= d >>> against your library, as the indication that your feature is supported ?= >>> If not, how does it differ from the presence of some additional note ? >>=20 >> If we can eliminate static linking for libxo, than >> that is definitely easy. Easiest probably. The >> question becomes: is it acceptable to not support >> static linking for libxo? Or alternatively, is it >> acceptable to not be able to check for the feature >> on a static executable? > Let me clarify. I do not suggest to use DT_NEEDED for libxo (?) as the > alternative feature test, either. I consider both the proposed note > and the presence of dependency on libxo.so as equally wrong ways to > determine the perceived support for specific output formats. My point > was that note is essentially same as DT_NEEDED tag, with the same set of > architectural problems, but it is easier to see them for DT_NEEDED. >=20 > Static binaries is just a detail, and yes, I consider static linking > as place where we cannot hold an ABI guarantee. The good example > was kse libpthread, which broke statically linked threaded binaries. >=20 > My objections against use of ELF could be summarized as following: > - ELF is a binary standard for image activation and runtime system, not > for the app level features. >=20 > - Attempt to shoe-horn the discussed feature into binary format makes > the indicator cut of the actual feature. It is like the mark on the > plastic can, which may have no relation to the can content. From > this PoV, the options or special env variables are at the right > place. >=20 > - It is unportable. Why bind the pure data transformation library to > the platform ? What about Mach-O or COFF/PE platforms ? What if > FreeBSD decide to change binary format, or add support for a new > format, besides ELF ? What if the tool and consumer are of different > ABI ? E.g., one is 64 bit, another is 32bit. Usually, there is > support for less bits, but 32bit libelf or debugger or libunwind > or whatever usually have troubles reading 64bit ELF objects. >=20 >>=20 >> For the first I'm inclined to say yes, but not for >> the second. >=20 > In fact, since I already started talking on the subject, I > dislike the idea of adding the other text formats to existing > tools. Much more reasonable route, IMO, is to have a library to > interrogate system state and present the data in ABI-stable and > introspection-friendly way, without enforcing the >=20 > kernel->internal userspace structure->text serialization-> > text parsing ->some other internal userspace structure >=20 > path on the consumers. >=20 > If so inclined, some existing tools could be converted to use the > library and possibly libxo. But I suspect that the presence of the > library for system state and bindings for the most popular scripting > languages (which is facilitated by the introspection facilities, > to avoid the need of translating each ABI structure into each language > datatype) is the sweet spot for the current consumers of libxo > formatted output. There would be no need for running external tool, > determining if it is suitable, parsing its output, treat the text > output as ABI contract and so on. From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 15:19:05 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 97D6E9D1; Fri, 15 Aug 2014 15:19:05 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 747872C6F; Fri, 15 Aug 2014 15:19:05 +0000 (UTC) Received: from [10.0.1.20] (c-76-21-10-192.hsd1.ca.comcast.net [76.21.10.192]) by elvis.mu.org (Postfix) with ESMTPSA id DCA85346DE23; Fri, 15 Aug 2014 08:19:04 -0700 (PDT) Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=windows-1252 From: Alfred Perlstein In-Reply-To: Date: Fri, 15 Aug 2014 08:19:12 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <3D4EDA4C-6D60-4D05-A1BD-44121EF020B5@mu.org> References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> <20140815082501.GR2737@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1283) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , "arch@freebsd.org" , Poul-Henning Kamp , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 15:19:05 -0000 On Aug 15, 2014, at 7:44 AM, Alfred Perlstein wrote: > Agree with kib.=20 >=20 > Kib, one thing that I just don't understand is the point of even being = able to query programs for libxo support. Does it make sense to you? =20 I guess one other thing I should mention is that I understand the point = of querying programs for support, but just not in this case. An example where this would make more sense would be for instance for = binaries that can run as a process thread without global state. An example of this would be a version of ls(1) or even awk(1) that = instead of the shell doing a fork/exec for a pipeline, it would note = that the program supports "threaded mode" and instead dlopen the program = and call a special function to run this former subprocess as a thread. This would reduce IPC cost and cost of fork/exec. In that case one = makes the assumption that if the program is somehow tagged then it can = be run as a thread regardless of command line args. However in the case of formatted output=85 I just don't see the need. Can someone explain an actual use case here that makes sense? -Alfred >=20 > Sent from my iPhone >=20 >> On Aug 15, 2014, at 1:25 AM, Konstantin Belousov = wrote: >>=20 >>> On Thu, Aug 14, 2014 at 10:04:36AM -0700, Marcel Moolenaar wrote: >>>=20 >>> On Aug 13, 2014, at 10:26 PM, Konstantin Belousov = wrote: >>>>>=20 >>>>> I know ELF "Note" elements can be used to carry vendor-specific >>>>> data, but have no experience with them. Would it be reasonable to >>>>> use them as a means of communicating this information to other = bits >>>>> of software? >>>> No. >>>=20 >>> Too extreme. >>>=20 >>>>> Is FreeBSD using Notes for other information currently? >>>> Yes, the notes are used to communicate the information required by >>>> the dynamic linker to correctly activate the image. The mechanism = has >>>> nothing to do with application-specific features, and overloading = it for >>>> that purpose is severe and pointless layering violation. Things = should >>>> not be done just because they could be done. >>>=20 >>> Too extreme. Life is a lot more subtle. Standards >>> are as well. There are many examples in the real >>> world where standards are interpreted a little >>> more liberal than others may want to. When such >>> result in (gratuitous) incompatibilities, we all >>> interpret it as bad. But when it adds real value, >>> you tend to find it in the next update of the >>> standard. >>=20 >> Do you suggest that next revision of ELF standard starts talking >> about app-level features ? >>=20 >>>=20 >>>> Using the static tagging for the dynamic application properties is = wrong >>>> anyway. E.g., would you consider the mere fact that the binary is = linked >>>> against your library, as the indication that your feature is = supported ? >>>> If not, how does it differ from the presence of some additional = note ? >>>=20 >>> If we can eliminate static linking for libxo, than >>> that is definitely easy. Easiest probably. The >>> question becomes: is it acceptable to not support >>> static linking for libxo? Or alternatively, is it >>> acceptable to not be able to check for the feature >>> on a static executable? >> Let me clarify. I do not suggest to use DT_NEEDED for libxo (?) as = the >> alternative feature test, either. I consider both the proposed note >> and the presence of dependency on libxo.so as equally wrong ways to >> determine the perceived support for specific output formats. My point >> was that note is essentially same as DT_NEEDED tag, with the same set = of >> architectural problems, but it is easier to see them for DT_NEEDED. >>=20 >> Static binaries is just a detail, and yes, I consider static linking >> as place where we cannot hold an ABI guarantee. The good example >> was kse libpthread, which broke statically linked threaded binaries. >>=20 >> My objections against use of ELF could be summarized as following: >> - ELF is a binary standard for image activation and runtime system, = not >> for the app level features. >>=20 >> - Attempt to shoe-horn the discussed feature into binary format makes >> the indicator cut of the actual feature. It is like the mark on the >> plastic can, which may have no relation to the can content. From >> this PoV, the options or special env variables are at the right >> place. >>=20 >> - It is unportable. Why bind the pure data transformation library to >> the platform ? What about Mach-O or COFF/PE platforms ? What if >> FreeBSD decide to change binary format, or add support for a new >> format, besides ELF ? What if the tool and consumer are of different >> ABI ? E.g., one is 64 bit, another is 32bit. Usually, there is >> support for less bits, but 32bit libelf or debugger or libunwind >> or whatever usually have troubles reading 64bit ELF objects. >>=20 >>>=20 >>> For the first I'm inclined to say yes, but not for >>> the second. >>=20 >> In fact, since I already started talking on the subject, I >> dislike the idea of adding the other text formats to existing >> tools. Much more reasonable route, IMO, is to have a library to >> interrogate system state and present the data in ABI-stable and >> introspection-friendly way, without enforcing the >>=20 >> kernel->internal userspace structure->text serialization-> >> text parsing ->some other internal userspace structure >>=20 >> path on the consumers. >>=20 >> If so inclined, some existing tools could be converted to use the >> library and possibly libxo. But I suspect that the presence of the >> library for system state and bindings for the most popular scripting >> languages (which is facilitated by the introspection facilities, >> to avoid the need of translating each ABI structure into each = language >> datatype) is the sweet spot for the current consumers of libxo >> formatted output. There would be no need for running external tool, >> determining if it is suitable, parsing its output, treat the text >> output as ABI contract and so on. > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to = "freebsd-arch-unsubscribe@freebsd.org" >=20 From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 15:30:47 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5024E74 for ; Fri, 15 Aug 2014 15:30:46 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B338D2E7A for ; Fri, 15 Aug 2014 15:30:46 +0000 (UTC) Received: from [192.168.2.22] (atc.xcllnt.net [50.0.150.213]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s7FFUUWJ008368 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 15 Aug 2014 08:30:30 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_B2A82E7C-1674-4971-8B4D-BDC20FE661A2"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: ELF and feature checking [was: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML] From: Marcel Moolenaar In-Reply-To: <20140815082501.GR2737@kib.kiev.ua> Date: Fri, 15 Aug 2014 08:30:29 -0700 Message-Id: <247A649F-0195-493A-A668-5D05F0B5100E@xcllnt.net> References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> <20140815082501.GR2737@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1878.6) Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 15:30:47 -0000 --Apple-Mail=_B2A82E7C-1674-4971-8B4D-BDC20FE661A2 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Aug 15, 2014, at 1:25 AM, Konstantin Belousov wrote: > My objections against use of ELF could be summarized as following: > - ELF is a binary standard for image activation and runtime system, not > for the app level features. I think you'll find that your definition is wrong from the outset. ELF is also the container for relocatable objects and intermediate compiler code when doing whole-program optimizations (i.e. the actual compilation happens at the link stage). It's also the container for memory snapshot and core dumps. Neither are actually needed to implement a runtime system. It's an added application level feature! ELF explicitly mentions some of those use cases, so it's not defined for just activation and runtime system. When a compiler puts intermediate code in an ELF container, we're already talking about app-level usage. It's been done for more than 10 years so you're way off in the left field on this one. Come join us :-) The reverse is also interest: On amd64 we use relocatable objects as kernel loadable modules. Heresy! > - Attempt to shoe-horn the discussed feature into binary format makes > the indicator cut of the actual feature. It is like the mark on the > plastic can, which may have no relation to the can content. From > this PoV, the options or special env variables are at the right > place. Sure. But why do you think grocery stores have shelves full of colored and labeled cans? We do need to be able to see from the can what it is we're putting in our cart, right? It would be a real mess if everyone starts opening cans left and right to find the can with the black beans and not the kidney beans. Software isn't that much different that the analogy doesn't apply. Being able to quick check the container for something is not unreasonable. Not being able to discuss such an option because people can't humor each other, is not good for the project. We'll get to a good solution in the end -- even if it's one you or I don't like. So lighten up. > - It is unportable. That's neither here nor there. On the one hand we have people say that we're over engineering and on the other hand we have people say that it's not portable. If those people get a room and make a baby, we'll have the balance we need! Why are people so afraid to explore? Why do people feel a need to pull the leash every time a discussion starts to wander. Innovation is found when the leash is extended and being pulled on... We're not looking for a portable solution. We are still discussing if it's doable or wanted. Portability comes into play when you have found one or more solutions. Without a solution portability is like an attribute without an object. -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_B2A82E7C-1674-4971-8B4D-BDC20FE661A2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPuJ5UACgkQpgWlLWHuifZVQACfY1BAqCht+n2noZs/6EZyc/LC +FsAnibEOBPh8k4ubQVsV3ZN8KkhIHk8 =HYU3 -----END PGP SIGNATURE----- --Apple-Mail=_B2A82E7C-1674-4971-8B4D-BDC20FE661A2-- From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 16:13:49 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEB57874; Fri, 15 Aug 2014 16:13:49 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0241.outbound.protection.outlook.com [207.46.163.241]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1CDA8254F; Fri, 15 Aug 2014 16:13:48 +0000 (UTC) Received: from BY2PR05CA062.namprd05.prod.outlook.com (10.141.250.52) by CO2PR05MB731.namprd05.prod.outlook.com (10.141.228.21) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Fri, 15 Aug 2014 16:13:46 +0000 Received: from BN1AFFO11FD029.protection.gbl (2a01:111:f400:7c10::146) by BY2PR05CA062.outlook.office365.com (2a01:111:e400:2c5f::52) with Microsoft SMTP Server (TLS) id 15.0.1010.18 via Frontend Transport; Fri, 15 Aug 2014 16:13:45 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1AFFO11FD029.mail.protection.outlook.com (10.58.52.184) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Fri, 15 Aug 2014 16:13:45 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 15 Aug 2014 09:13:43 -0700 Received: from idle.juniper.net (idleski.juniper.net [172.25.4.26]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7FGDWn17204; Fri, 15 Aug 2014 09:13:33 -0700 (PDT) (envelope-from phil@juniper.net) Received: from idle.juniper.net (localhost [127.0.0.1]) by idle.juniper.net (8.14.4/8.14.3) with ESMTP id s7FGDMmt003567; Fri, 15 Aug 2014 12:13:22 -0400 (EDT) (envelope-from phil@idle.juniper.net) Message-ID: <201408151613.s7FGDMmt003567@idle.juniper.net> To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <3D4EDA4C-6D60-4D05-A1BD-44121EF020B5@mu.org> Date: Fri, 15 Aug 2014 12:13:22 -0400 From: Phil Shafer MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199003)(164054003)(20776003)(21056001)(68736004)(46102001)(95666004)(47776003)(64706001)(110136001)(50466002)(74662001)(99396002)(85852003)(105596002)(54356999)(53416004)(87936001)(103666002)(107046002)(81542001)(106466001)(81156004)(102836001)(86362001)(77982001)(81342001)(69596002)(76506005)(84676001)(4396001)(44976005)(31966008)(48376002)(6806004)(92566001)(79102001)(50986999)(76482001)(80022001)(97736001)(85306004)(83072002)(74502001)(92726001)(83322001); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB731; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 0304E36CA3 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=phil@juniper.net; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , "arch@freebsd.org" , Poul-Henning Kamp , Konstantin Belousov , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 16:13:50 -0000 Alfred Perlstein writes: >Can someone explain an actual use case here that makes sense? In JUNOS, we support a NETCONF API, allowing NETCONF RPCs (in XML) to get hierarchical data back (in XML). We use this to automate management of our devices. When we parse RPCs, we construct command lines that are invoked. For example the "show interfaces terse" command in in the CLI is available as the RPC with the option. The JUNOS CLI parses either of these into the comand line "ifinfo -b". We currently are told which commands support XML output and which don't. For those that do, we simply forward the command's output to the client. For those that don't we wrap the output in an XML tag that means "we don't support this in XML yet, but here's the text" (and escape the data). The benefit of this intrastructure is that clients can parse and digest the data easier in XML, and can perform scripts like: for $if in ($res/interfaces/interface[starts-with(name, "xe-")]) { if ($if/gigether-options/loopback) { call warning($if, "interface is in loopback mode"); } } libxo will add a third class where we set LIBXO_OPTIONS to trigger XML output (and hopefully eliminate the first one, where we are currently using the "-X" option to tell commands to make XML output). I can also imagine a similar server scenario for the restful server using JSON. Personally, I'm not a fan of JSON, but it's popular enough to make it a required output style, given that all styles are generated by the same xo_emit calls. In the more science fiction work, I can imagine using the HTML output as a means to escape the 70's era tty world, running a webkit-enabled shell that can trigger HTML output for libxo-enabled binaries and decorate the output with custom HTML5 constructs. My theory is that keyboards are reasonable input devices, but ttys are poor output devices. With JUNOS, we have a CLI that runs over tty/ssh/etc and gives command line access to the device. We have on-box scripting for extensibility, but are hindered by the lack of features in display output. I tried a number of times to make a decent looking ascii sparkline before giving up. If I can marry the high-bandwidth input capabilities of the input with the high-capacity output of the HTML5 world, I can have a world where I can combine the flexibility and power that keep me in the tty world with the direct manipulation and display technology that makes the HTML5 world appealing. So a shell that understands that some commands can generate HTML, notices when they are the last member of a pipeline, and knows to trigger them to make HTML is a useful first step. (Well, actually the first step will likely be a "shell in a browser window" sort of thing, but....) In any case, the first round of any of these technologies can get by initially with explicit knowledge of what does or does support libxo. If/when this actually works, we can revisit it. Thanks, Phil From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 17:39:32 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 000EA560; Fri, 15 Aug 2014 17:39:31 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0189.outbound.protection.outlook.com [207.46.163.189]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F3DB420A2; Fri, 15 Aug 2014 17:39:30 +0000 (UTC) Received: from BL2PR05CA0016.namprd05.prod.outlook.com (10.255.226.16) by BY2PR05MB728.namprd05.prod.outlook.com (10.141.223.25) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Fri, 15 Aug 2014 17:39:21 +0000 Received: from BN1BFFO11FD012.protection.gbl (2a01:111:f400:7c10::1:143) by BL2PR05CA0016.outlook.office365.com (2a01:111:e400:c04::16) with Microsoft SMTP Server (TLS) id 15.0.1010.18 via Frontend Transport; Fri, 15 Aug 2014 17:39:20 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BN1BFFO11FD012.mail.protection.outlook.com (10.58.144.75) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Fri, 15 Aug 2014 17:39:20 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 15 Aug 2014 10:38:32 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7FHcUn78640; Fri, 15 Aug 2014 10:38:31 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 93832580A2; Fri, 15 Aug 2014 10:38:30 -0700 (PDT) To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <53EDB0EF.6090902@mu.org> References: <201408141640.s7EGe422096656@idle.juniper.net> <53ED57F2.5020808@mu.org> <20140815053604.9E40B580A2@chaos.jnpr.net> <53EDB0EF.6090902@mu.org> Comments: In-reply-to: Alfred Perlstein message dated "Fri, 15 Aug 2014 00:04:15 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 15 Aug 2014 10:38:30 -0700 Message-ID: <20140815173830.93832580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199003)(189002)(21056001)(92566001)(106466001)(88136002)(86362001)(81342001)(62966002)(57986006)(90896003)(105596002)(69596002)(4396001)(85306004)(46102001)(104166001)(83072002)(76506005)(77982001)(107046002)(93916002)(102176002)(93886004)(50226001)(76176999)(102836001)(70486001)(50986999)(74662001)(48376002)(81542001)(68736004)(31966008)(6806004)(74502001)(20776003)(81156004)(44976005)(87286001)(80022001)(87936001)(76482001)(33656002)(64706001)(95666004)(85852003)(47776003)(110136001)(83322001)(79102001)(92726001)(99396002)(50466002)(97736001)(77156001)(84676001)(101356003)(93546004)(89996001)(42262002); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB728; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 0304E36CA3 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 17:39:32 -0000 On Fri, 15 Aug 2014 00:04:15 -0700, Alfred Perlstein writes: >Sure that is fairly simple, but where will that really even be needed in >practice right now? You did work at Juniper for a while - did you never use Junos? If you had you should be well aware of at least one use case. No one said appA needs to be a standard bsd utility. >How many programs have been successfully converted over to libxo at this >point? How is that relevant to any of this discussion? From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 18:37:52 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4286CD4D; Fri, 15 Aug 2014 18:37:52 +0000 (UTC) Received: from smtp9.server.rpi.edu (smtp9.server.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D3F3928A1; Fri, 15 Aug 2014 18:37:51 +0000 (UTC) Received: from smtp-auth2.server.rpi.edu (smtp-auth2.server.rpi.edu [128.113.2.232]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s7FIRfwm013944; Fri, 15 Aug 2014 14:27:41 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id 2AE0F180E9; Fri, 15 Aug 2014 14:27:41 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id B2A621802B; Fri, 15 Aug 2014 14:27:40 -0400 (EDT) From: "Garance A Drosehn" To: "Alfred Perlstein" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Fri, 15 Aug 2014 14:27:39 -0400 Message-ID: <2B9447E2-AF2F-4B8F-8D6F-2A9F2AF9AB10@rpi.edu> In-Reply-To: References: <201408131936.s7DJaA1r089174@idle.juniper.net> <20140814052648.GM2737@kib.kiev.ua> <16CE0543-0616-415D-9E68-EEB053DE4254@xcllnt.net> <20140815082501.GR2737@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 02MD6rFSS X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , "arch@freebsd.org" , Poul-Henning Kamp , Konstantin Belousov , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 18:37:52 -0000 [apologies to Alfred, who will be seeing this twice...] On 15 Aug 2014, at 10:44, Alfred Perlstein wrote: > > Agree with kib. > > Kib, one thing that I just don't understand is the point of even being > able to query programs for libxo support. Does it make sense to you? If I'm writing cross-platform scripts, then I'll want some way for the script to know which commands have machine-readable output, and which ones do not. And unless we're going to implement machine-readable output in all base-system commands at once, then we're also going to have the issue that a command which does not do command-readable output today will suddenly start doing it tomorrow. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 18:46:16 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A673CD6; Fri, 15 Aug 2014 18:46:16 +0000 (UTC) Received: from smtp10.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 570A229F0; Fri, 15 Aug 2014 18:46:15 +0000 (UTC) Received: from smtp-auth1.server.rpi.edu (route.canit.rpi.edu [128.113.2.231]) by smtp10.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s7FIjg9N005936; Fri, 15 Aug 2014 14:45:43 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id DD465580CD; Fri, 15 Aug 2014 14:45:42 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id BA06D580B7; Fri, 15 Aug 2014 14:45:42 -0400 (EDT) From: "Garance A Drosehn" To: "Marcel Moolenaar" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Fri, 15 Aug 2014 14:45:41 -0400 Message-ID: In-Reply-To: <613EB1A5-2932-446A-A9A2-8CBDD060A00B@xcllnt.net> References: <201408131936.s7DJaA1r089174@idle.juniper.net> <613EB1A5-2932-446A-A9A2-8CBDD060A00B@xcllnt.net> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 03MD6JHXh X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.230 Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 18:46:16 -0000 On 13 Aug 2014, at 19:02, Marcel Moolenaar wrote: > On Aug 13, 2014, at 12:36 PM, Phil Shafer wrote: >> >> I've a related topic: when an app goes to run a child command, how >> can it determine whether that binary supports libxo-based encoding >> requests? This should be known before the binary is run, since >> there's no means of auto-detecting the supported output after the >> fact. >> >> For example, say I want to make a JSON-based API for my server. I >> can setenv("LIBXO_OPTIONS", "json") to get JSON output, but I won't >> know if the binary supports this or if the output needs to be wrapped >> and escaped. > > Aside: > Using environment variables can be handy, but isn't always. > What do you think about calling a libxo init function from > main() and giving it argc and argv so that libxo options > are parsed and removed just like what xlib does? Here are two alternate ideas for controlling the machine-readable output, in a way where a script can easily tell *if* a command can generate machine-readable output, and we can provide some set of common arguments across all commands which do support that output. 1. For each command, the command could check what name it is started with. If the main() routine sees the command was started as 'xo-netstat', then it generates machine-readable output and supports some extra xo-specific arguments. It is pretty easy to come up with some workable syntax for those arguments if the command has a new command name. 2. some shell for launching an xo-aware command. In this case only one command would be added to the system, and you'd say 'xocmd netstat ...' instead of 'xo-netstat ...'. The special xo-specific parameters would be specified between the 'xocmd' and the standard unix command ('netstat', in this example). If you're wondering how this could be done: How about having each command looking for a weak-external symbol, and it generates the machine-readable output if that loader symbol was resolved when the command started up. Maybe that loader could be the entry point for a large chunk of the libxo library. Thus a unix command running without the libxo support would be only slightly bigger than it is now. Please note that in both cases, I am not saying that anyone would write two separate versions of each unix command. I'm saying = there would still be just one set of source files per unix command, and each unix command would decide which type of output to generate based on how it was launched. -- = Garance Alistair Drosehn =3D drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 18:30:50 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E4977B46; Fri, 15 Aug 2014 18:30:49 +0000 (UTC) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8580427A5; Fri, 15 Aug 2014 18:30:49 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id wp18so2271265obc.20 for ; Fri, 15 Aug 2014 11:30:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qipc5u84yHxJ7u7jQDcPxvvIexmFYvWzbku14PimcjA=; b=MVGW5tD8mfq4nhulIxayIopxufaMlSvqXR6ao5/Ym42urop3vF+WdW5DanRTwXh7Ya VD+a3gldabD9wa94qcqX2JaoupYsJuerbPEvTVxOslbYKl4TtfIOk3RkhMJerVo1fEla JKSSJooQI79dsUFzfGMuO+CfVKEY+qYPE2u3iq+dLVxqrwpkjPlHXE3OtVuSYaWFHxbK HZY7h+NBqiw6BYw7r5HdtlnDbZqXaF7LpLF8/rvCsEFXnenAusg6bLonEiqega2HK26Q 1KzWVh+rxj+aeUDNMZHry+quG7IN3xIZZvPO1Vg4GbPHphSf2VP/en/D0stU3dB7W7nd dcYA== MIME-Version: 1.0 X-Received: by 10.182.60.4 with SMTP id d4mr22202771obr.4.1408127448848; Fri, 15 Aug 2014 11:30:48 -0700 (PDT) Received: by 10.182.216.200 with HTTP; Fri, 15 Aug 2014 11:30:48 -0700 (PDT) In-Reply-To: <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> <201408140847.00573.jhb@freebsd.org> <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> Date: Fri, 15 Aug 2014 20:30:48 +0200 Message-ID: Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML From: Oliver Pinter To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 X-Mailman-Approved-At: Fri, 15 Aug 2014 18:52:46 +0000 Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , "Simon J. Gerraty" , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 18:30:50 -0000 On 8/14/14, Warner Losh wrote: > > On Aug 14, 2014, at 6:47 AM, John Baldwin wrote: > >>>> Marking the binary with a libxo-specific note tells the caller that >>>> the binary is capable of rendering its output in a non-traditional >>>> style and gives the caller a means of triggering those styles of >>>> output. In the libxo-enabled world, I see this as vital information >>>> the caller needs to initialize the environment in which the command >>>> will be run. Isn't this exactly the sort of information ELF targets >>>> for note sections? >>> >>> How binary format has any relevance for an application level feature ? >>> What would you do with the binaries which permissions are 'r-s--x--x', >>> which is not unexpected for the tools which gather system information >>> and have to access things like /dev/mem ? >>> >>> You removed and did not answered a crusial question, which is a litmus >>> test for your proposal. Namely, how presence of the proposed note in >>> the binary is different from DT_NEEDED tag for your library ? >> >> Yes, checking DT_NEEDED for libxo.so is the first thing I thought of as >> well. >> It is equivalent to 'ldd foo | grep libxo'. > > Doesn't work for static binaries, nor for cases where libxo is linked in by > a > library indirectly, nor for when the command is a shell script that may > invoke a command that supports this output, nor for a python script that > implements this output, etc. > > My question for people advocating this method: Why not require all commands > that generate this kind of output to support a standard command line option > that causes the command to print nothing and return 0 if it supports > reporting, > or anything else if it doesn't (return 0 with output, or return non-zero > with or without > output). This would handle the more complicated implementation issues with > using > DT_NEEDED and/or the ELF note, be more in line with how things are > traditionally > done, and offer greater flexibility of implementation. > > Warner What's about extending standard file descriptors? Example by: 0: stdin 1: stdout 2: stderr N: extended stdin N+1: extended stdout N+2: extended stderr If the program can not open fd [0-2], then in fallback case can try to open fd [N-N+2]. > From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 19:13:23 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A18CDD79; Fri, 15 Aug 2014 19:13:23 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7756F2DC5; Fri, 15 Aug 2014 19:13:23 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id DBDE7B963; Fri, 15 Aug 2014 15:13:20 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: [PATCH 0/2] plug capability races Date: Fri, 15 Aug 2014 10:31:45 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> In-Reply-To: <1408064112-573-1-git-send-email-mjguzik@gmail.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408151031.45967.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 15 Aug 2014 15:13:21 -0400 (EDT) Cc: Konstantin Belousov , Mateusz Guzik , Robert Watson , Johan Schuijt X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 19:13:23 -0000 On Thursday, August 14, 2014 8:55:10 pm Mateusz Guzik wrote: > fget_unlocked currently reads 'fde' which is a structure consisting of > serveral fields. In effect the read is inatomic and may result in > obtaining file pointer with stale or incorrect capabilities. > > Example race is with dup2. > > Side effect is that capability checks can be circumvented. > > Proposed way to fix it is with the help of sequence counters. > > Patchset assumes stuff from > 'Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked' > ( http://lists.freebsd.org/pipermail/freebsd-arch/2014-July/015550.html ) > is applied. There is no technical dependency between patches (apart from > READ_ONCE), but this patch amortizes performance hit introduced with seqlock. > > So this introduces a measurable hit with a microbenchmark (16 threads > reading from a pipe which fails with EAGAIN), but is still much faster than > current code with atomic_load_acq_int(&fdp->fd_nfiles). > > x propernoacq-readpipe-run-sum > + seq2-noacq-readpipe-run-sum > N Min Max Median Avg Stddev > x 20 59479718 59527286 59496714 59499504 13752.968 > + 20 54520752 54920054 54829539 54773480 136842.96 > Difference at 95.0% confidence > -4.72602e+06 +/- 62244.4 > -7.94296% +/- 0.104613% > (Student's t, pooled s = 97250) > > There is still one theoretical race unfixed, but I don't believe it matters > much. > > The race is: > fp gets reallocated before refcount check. this resuls in returning fp > regardless of new caps, but I don't see how this particular race could be > exploited. It could be fixed by re-reading entire fde and checking if it > changed. One thing I would like to see is for the timecounter code to be adapted to use the seq API instead of doing it by hand (the timecounter code is also missing barriers due to doing it by hand). Also, seq needs a seq(9) manpage. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 19:42:45 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B75AC690 for ; Fri, 15 Aug 2014 19:42:45 +0000 (UTC) Received: from smtp9.server.rpi.edu (smtp9.server.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 839A22162 for ; Fri, 15 Aug 2014 19:42:45 +0000 (UTC) Received: from smtp-auth1.server.rpi.edu (route.canit.rpi.edu [128.113.2.231]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id s7FJghWB026418 for ; Fri, 15 Aug 2014 15:42:43 -0400 Received: from smtp-auth1.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth1.server.rpi.edu (Postfix) with ESMTP id AC7FA580DE for ; Fri, 15 Aug 2014 15:42:43 -0400 (EDT) Received: from [128.113.24.47] (gilead-qc124.netel.rpi.edu [128.113.124.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: drosih) by smtp-auth1.server.rpi.edu (Postfix) with ESMTPSA id 9DF2A580CD for ; Fri, 15 Aug 2014 15:42:43 -0400 (EDT) From: "Garance A Drosehn" To: arch@freebsd.org Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Date: Fri, 15 Aug 2014 15:42:41 -0400 Message-ID: <5B031AE6-100F-4AD7-B721-F0FF207CAE3E@rpi.edu> In-Reply-To: <53ED578B.6070205@freebsd.org> References: <20140814052648.GM2737@kib.kiev.ua> <201408140606.s7E66XXA091972@idle.juniper.net> <20140814085257.GN2737@kib.kiev.ua> <201408140847.00573.jhb@freebsd.org> <94A47A7D-89C9-4504-B669-2A5EDA80373B@bsdimp.com> <53ED578B.6070205@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailer: MailMate (1.7.2r3905) X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 15.10] X-CanIt-Incident-Id: 02MD7GHts X-CanIt-Geo: ip=128.113.124.17; country=US; region=New York; city=Troy; latitude=42.7495; longitude=-73.5951; http://maps.google.com/maps?q=42.7495,-73.5951&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 19:42:45 -0000 On 14 Aug 2014, at 20:42, Alfred Perlstein wrote: > On 8/14/14 9:13 AM, Warner Losh wrote: >> My question for people advocating this method: Why not require all >> commands >> that generate this kind of output to support a standard command line >> option >> that causes the command to print nothing and return 0 if it supports >> reporting, >> or anything else if it doesn’t (return 0 with output, or return >> non-zero with or without >> output). This would handle the more complicated implementation issues >> with using >> DT_NEEDED and/or the ELF note, be more in line with how things are >> traditionally >> done, and offer greater flexibility of implementation. >> >> Warner > This is a decent idea, however the problem is firstly that most > short-opts are taken, second issue is that adding getopt_long to a > whole slew of programs will make the effort take a long time. It's > really better to limit scope of this such that we are just making > machine readable output. > > -Alfred [-- try #2 with this reply. arg. --] In the case of adding just one long option, you don't have to add getopt_long() to any programs. The main program would only need: /* - - - - - - - - - - - - - - - - - - - */ /* Apologies if my email client totally mangles the following source */ #include #include #include #define COMMON_LONG_ARG "-—supports-xml" int main(int argc, char **argv) { /* note this whole check could be a macro pulled from some * include file */ if (argc == 2) { argv++; if (0 == strcmp(*argv, COMMON_LONG_ARG)) { return (0); } } printf("... Standard processing of %d arguments...\n", argc); /* I'm returning "1" here to mimic what would happen * in a command which did *not* have the above check. */ return (1); } /* - - - - - - - - - - - - - - - - - - - */ I think we could do better than the suggestion Warner gave, but his suggestion is pretty trivial to implement. It gets a little messier if we want to support multiple xo-related parameters, but again we could put all that standard processing in a library subroutine, and each program which has libxo would call that standard libxo-options routine before calling anything else in the unix command. -- Garance Alistair Drosehn = drosih@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Fri Aug 15 23:25:16 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CDFE87E0; Fri, 15 Aug 2014 23:25:16 +0000 (UTC) Received: from dmz-mailsec-scanner-3.mit.edu (dmz-mailsec-scanner-3.mit.edu [18.9.25.14]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 549B92A9C; Fri, 15 Aug 2014 23:25:15 +0000 (UTC) X-AuditID: 1209190e-f79946d000007db1-e1-53ee95a6ef85 Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP id F7.5F.32177.6A59EE35; Fri, 15 Aug 2014 19:20:06 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id s7FNK6pu029459; Fri, 15 Aug 2014 19:20:06 -0400 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s7FNK4Pm020246 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 15 Aug 2014 19:20:05 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id s7FNK3Zf028276; Fri, 15 Aug 2014 19:20:03 -0400 (EDT) Date: Fri, 15 Aug 2014 19:20:03 -0400 (EDT) From: Benjamin Kaduk X-X-Sender: kaduk@multics.mit.edu To: Mateusz Guzik Subject: Re: current fd allocation idiom In-Reply-To: <20140813015627.GC17869@dft-labs.eu> Message-ID: References: <20140717235538.GA15714@dft-labs.eu> <20140718155959.GN93733@kib.kiev.ua> <20140718191928.GB7179@dft-labs.eu> <201408111124.52064.jhb@freebsd.org> <20140812233617.GA17869@dft-labs.eu> <20140813015627.GC17869@dft-labs.eu> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkleLIzCtJLcpLzFFi42IRYrdT0V0+9V2wwR0+i9nTpzFZ7D1yndmi YdpjNovGg4tZHFg8Znyaz+Kxc9Zd9gCmKC6blNSczLLUIn27BK6MmW9CCiZzVsz9+I2tgbGd vYuRg0NCwERi2QqWLkZOIFNM4sK99WxdjFwcQgKzmSSu9m9gh3A2MkosXvmDBcI5xCSxrG8p M4TTwCix7eJPJpB+FgFtidNnpoDZbAJqEo/3NrNCzFWU2HxqEjOILSKgKvH86HqwOLNAgcSu Xe/B4sICGhJ3dq4Cu4NTwFDi0JXzYHFeAUeJ0+/3gMWFBNYzSWz44g9iiwroSKzeP4UFokZQ 4uTMJywQM7Uklk/fxjKBUWgWktQsJKkFjEyrGGVTcqt0cxMzc4pTk3WLkxPz8lKLdI31cjNL 9FJTSjcxgkKaU5JvB+PXg0qHGAU4GJV4eAXE3wULsSaWFVfmHmKU5GBSEuUNbgcK8SXlp1Rm JBZnxBeV5qQWH2KU4GBWEuFd5QaU401JrKxKLcqHSUlzsCiJ8761tgoWEkhPLEnNTk0tSC2C ycpwcChJ8M6dAtQoWJSanlqRlplTgpBm4uAEGc4DNPwOSA1vcUFibnFmOkT+FKOilDjvU5CE AEgiozQPrheWcl4xigO9Isx7FKSKB5iu4LpfAQ1mAhpcs/ktyOCSRISUVAPjkv6kJ3f/d7Ab MvIvXcRo5qu73qHW8TS/d0vZqcbsGsF98gwazjvc3n7Uuryo44Xe2qs32Z6cEZ//aPex/Z9Y q13VGFLOrM6I3XW1S7jowGZ+7Tf6C2Q+Pt7T9OSAs+/J8OuCX7fkKzzVXDDh8SM/r4Ky06U3 niYIVER22E9Vv2LtyzzjRMBhJZbijERDLeai4kQAYpBHOxQDAAA= Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2014 23:25:17 -0000 On Tue, 12 Aug 2014, Mateusz Guzik wrote: > On Tue, Aug 12, 2014 at 09:31:15PM -0400, Benjamin Kaduk wrote: > > On Tue, Aug 12, 2014 at 7:36 PM, Mateusz Guzik wrote: > > > > > I would expect soabort to result in a timeout/reset as opposed to regular > > > connection close. > > > > > > Comments around soabort suggest it should not be used as a replacement > > > for close, but maybe this is largely because of what the other end will > > > see. That will need to be investigated. > > > > > > > > I added some text regarding soabort to socket.9 in r266962 -- does that > > help clarify the situation? > > > > Nope. :-) > > It is unclear if the only motivation here is making sure nobody else > sees the socket when given thread calls soabort. This would be easily > guaranteed in here: fd allocation failed, fp with given socket was never > exposed to anyone. > > So, if you say soabort would work here just fine, I'm happy to use it > and blame you for problems. :-) Hmm, I was hoping that jhb would chime in and save me from being on the hook, but it does look like soabort() would be acceptable in this case. -Ben From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 00:29:39 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B3E42EF2; Sat, 16 Aug 2014 00:29:39 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9317D20C5; Sat, 16 Aug 2014 00:29:39 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id AF0C7346DDEB; Fri, 15 Aug 2014 17:29:38 -0700 (PDT) Message-ID: <53EEA655.1070009@mu.org> Date: Fri, 15 Aug 2014 17:31:17 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Phil Shafer Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408151613.s7FGDMmt003567@idle.juniper.net> In-Reply-To: <201408151613.s7FGDMmt003567@idle.juniper.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Marcel Moolenaar , John-Mark Gurney , "Simon J. Gerraty" , "arch@freebsd.org" , Poul-Henning Kamp , Konstantin Belousov , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 00:29:39 -0000 Phil, thank you for explaining, comments inline. On 8/15/14 9:13 AM, Phil Shafer wrote: > Alfred Perlstein writes: >> Can someone explain an actual use case here that makes sense? > In JUNOS, we support a NETCONF API, allowing NETCONF RPCs (in XML) > to get hierarchical data back (in XML). We use this to automate > management of our devices. When we parse RPCs, we construct command > lines that are invoked. > > For example the "show interfaces terse" command in in the CLI is > available as the RPC with the > option. The JUNOS CLI parses either of these into the comand line > "ifinfo -b". > > We currently are told which commands support XML output and which > don't. For those that do, we simply forward the command's output > to the client. For those that don't we wrap the output in an XML > tag that means "we don't support this in XML yet, but here's the > text" (and escape the data). > > The benefit of this intrastructure is that clients can parse and > digest the data easier in XML, and can perform scripts like: > > for $if in ($res/interfaces/interface[starts-with(name, "xe-")]) { > if ($if/gigether-options/loopback) { > call warning($if, "interface is in loopback mode"); > } > } > > libxo will add a third class where we set LIBXO_OPTIONS to trigger > XML output (and hopefully eliminate the first one, where we are > currently using the "-X" option to tell commands to make XML output). OK, this is making sense now. It really is to "run any command" in almost a file browser sort of manner. Almost like OS X ".app" directories or embedding the Windows icon inside of a .exe. > > I can also imagine a similar server scenario for the restful server > using JSON. Personally, I'm not a fan of JSON, but it's popular > enough to make it a required output style, given that all styles > are generated by the same xo_emit calls. JSON would be huge, every time I mention that FreeBSD is about to grow this support the eyes of webapp developers and devops people light up with joy. > > In the more science fiction work, I can imagine using the HTML > output as a means to escape the 70's era tty world, running a > webkit-enabled shell that can trigger HTML output for libxo-enabled > binaries and decorate the output with custom HTML5 constructs. > > My theory is that keyboards are reasonable input devices, but ttys > are poor output devices. With JUNOS, we have a CLI that runs over > tty/ssh/etc and gives command line access to the device. We have > on-box scripting for extensibility, but are hindered by the lack > of features in display output. I tried a number of times to make > a decent looking ascii sparkline before giving up. > > If I can marry the high-bandwidth input capabilities of the input > with the high-capacity output of the HTML5 world, I can have a world > where I can combine the flexibility and power that keep me in the > tty world with the direct manipulation and display technology that > makes the HTML5 world appealing. > > So a shell that understands that some commands can generate HTML, > notices when they are the last member of a pipeline, and knows to > trigger them to make HTML is a useful first step. (Well, actually > the first step will likely be a "shell in a browser window" sort > of thing, but....) > > In any case, the first round of any of these technologies can get > by initially with explicit knowledge of what does or does support > libxo. If/when this actually works, we can revisit it. > Thank you, this makes a lot of sense now. I do think that maybe we ought to do a first pass with a mind for just support of the output formats and get as many utils moved over asap as opposed to wrapping this up in a larger architectural movement. Just FYI, using ELF notes does cause issues for scripts, unless perhaps those scripts have another way to signify that they will produce "xo-like" output. I'm just nervous that this might not get done because we get into deeper issues like the ELF sections as opposed to just making a good idea happen (libxo itself.). -Alfred From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 00:33:45 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61B4B7E; Sat, 16 Aug 2014 00:33:45 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4A6342164; Sat, 16 Aug 2014 00:33:45 +0000 (UTC) Received: from u10-2-16-021.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id ED57D346DE11; Fri, 15 Aug 2014 17:33:44 -0700 (PDT) Message-ID: <53EEA74B.9070107@mu.org> Date: Fri, 15 Aug 2014 17:35:23 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: "Simon J. Gerraty" Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML References: <201408141640.s7EGe422096656@idle.juniper.net> <53ED57F2.5020808@mu.org> <20140815053604.9E40B580A2@chaos.jnpr.net> <53EDB0EF.6090902@mu.org> <20140815173830.93832580A2@chaos.jnpr.net> In-Reply-To: <20140815173830.93832580A2@chaos.jnpr.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 00:33:45 -0000 On 8/15/14 10:38 AM, Simon J. Gerraty wrote: > On Fri, 15 Aug 2014 00:04:15 -0700, Alfred Perlstein writes: >> Sure that is fairly simple, but where will that really even be needed in >> practice right now? > You did work at Juniper for a while - did you never use Junos? > If you had you should be well aware of at least one use case. > No one said appA needs to be a standard bsd utility. Well I sort of used Junos, as you know during my tenure I threaded the JUNOS kernel for our platform and did some interesting realtime-ish tweaks. Unfortunately did not spend a lot of time in userland or using the box in the role of a customer. >> How many programs have been successfully converted over to libxo at this >> point? > How is that relevant to any of this discussion? > Well, it speaks towards the vision of getting this done in a timely manner. As I said there is a GSOC project that has a ton of code already done. If this libxo is ready to go in, it should go in and we should get towards converting more utils to using it. However if we are going to perpetually add frameworky things, but not convert over userland tools to the actual framework, then that is a potential problem worth calling out. -Alfred From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 01:29:53 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E84DC61; Sat, 16 Aug 2014 01:29:53 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C3C1264C; Sat, 16 Aug 2014 01:29:52 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id d1so1465876wiv.9 for ; Fri, 15 Aug 2014 18:29:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=2d1GCBETo3RGJRh9TopKYGpikDARqCyPcx6tPmJ3BdA=; b=kE4w8FrjOwMG7KVI0XLQzABkiXQiuRSumj2tDsifzzrrf7HAFELSRqFGr3U10vCVKw Yv1hP/cqyddVk3vlSGV0jhjzA0lE9f4Ku+KplohBa1Dma1OfGuamGLF2D/Qbmc9NZVCT hqxGXsvoHG8OIRJCk0Do1TEhl4a/+KTRH3r5LOuQksfrjetlk4sIhkNUs3Mh5v2LxAFQ jDLI2xh9YNuNDdB1CNQtLao6euHBKgvmnP3UcdRZrB+jdEjYRuCLyujAqpaZ3vr8WzTx uqXW/RxM7WEhYrHKDylvUGLJJltMS6/B0YblBIN/A6m8vBSmDNuNNdlCKZQBEKoA2oLj 9rbg== X-Received: by 10.194.205.129 with SMTP id lg1mr23223568wjc.97.1408152590067; Fri, 15 Aug 2014 18:29:50 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id e3sm22822029wjp.4.2014.08.15.18.29.48 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Fri, 15 Aug 2014 18:29:49 -0700 (PDT) Date: Sat, 16 Aug 2014 03:29:46 +0200 From: Mateusz Guzik To: John Baldwin Subject: Re: [PATCH 0/2] plug capability races Message-ID: <20140816012946.GA19135@dft-labs.eu> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <201408151031.45967.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201408151031.45967.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , Robert Watson , Johan Schuijt , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 01:29:53 -0000 On Fri, Aug 15, 2014 at 10:31:45AM -0400, John Baldwin wrote: > On Thursday, August 14, 2014 8:55:10 pm Mateusz Guzik wrote: > > fget_unlocked currently reads 'fde' which is a structure consisting of > > serveral fields. In effect the read is inatomic and may result in > > obtaining file pointer with stale or incorrect capabilities. > > > > Example race is with dup2. > > > > Side effect is that capability checks can be circumvented. > > > > Proposed way to fix it is with the help of sequence counters. > > > > Patchset assumes stuff from > > 'Getting rid of atomic_load_acq_int(&fdp->fd_nfiles)) from fget_unlocked' > > ( http://lists.freebsd.org/pipermail/freebsd-arch/2014-July/015550.html ) > > is applied. There is no technical dependency between patches (apart from > > READ_ONCE), but this patch amortizes performance hit introduced with > seqlock. > > > > So this introduces a measurable hit with a microbenchmark (16 threads > > reading from a pipe which fails with EAGAIN), but is still much faster than > > current code with atomic_load_acq_int(&fdp->fd_nfiles). > > > > x propernoacq-readpipe-run-sum > > + seq2-noacq-readpipe-run-sum > > N Min Max Median Avg Stddev > > x 20 59479718 59527286 59496714 59499504 13752.968 > > + 20 54520752 54920054 54829539 54773480 136842.96 > > Difference at 95.0% confidence > > -4.72602e+06 +/- 62244.4 > > -7.94296% +/- 0.104613% > > (Student's t, pooled s = 97250) > > > > There is still one theoretical race unfixed, but I don't believe it matters > > much. > > > > The race is: > > fp gets reallocated before refcount check. this resuls in returning fp > > regardless of new caps, but I don't see how this particular race could be > > exploited. It could be fixed by re-reading entire fde and checking if it > > changed. > > One thing I would like to see is for the timecounter code to be adapted to use > the seq API instead of doing it by hand (the timecounter code is also missing > barriers due to doing it by hand). > > Also, seq needs a seq(9) manpage. > Sure, I'm happy to write one later. Just in case I would like to note the following: There are some similarities to linux version of the same idea: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/seqlock.h On the other hand I don't see how one could implement this in a vastly different matter. -- Mateusz Guzik From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 02:07:58 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C931F19; Sat, 16 Aug 2014 02:07:58 +0000 (UTC) Received: from mail106.syd.optusnet.com.au (mail106.syd.optusnet.com.au [211.29.132.42]) by mx1.freebsd.org (Postfix) with ESMTP id C993B2911; Sat, 16 Aug 2014 02:07:57 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail106.syd.optusnet.com.au (Postfix) with ESMTPS id 618043C1C46; Sat, 16 Aug 2014 11:35:00 +1000 (EST) Date: Sat, 16 Aug 2014 11:34:59 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: John Baldwin Subject: Re: [PATCH 0/2] plug capability races In-Reply-To: <201408151031.45967.jhb@freebsd.org> Message-ID: <20140816102840.V1007@besplex.bde.org> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <201408151031.45967.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=fvDlOjIf c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=tTSYktBZc9AA:10 a=KN91Z2BipYgA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=vlCwttaAoV0VwkxwwJ0A:9 a=CjuIK1q_8ugA:10 Cc: Robert Watson , Mateusz Guzik , Konstantin Belousov , Johan Schuijt , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 02:07:58 -0000 On Fri, 15 Aug 2014, John Baldwin wrote: > One thing I would like to see is for the timecounter code to be adapted to use > the seq API instead of doing it by hand (the timecounter code is also missing > barriers due to doing it by hand). Locking in the timecounter code is poor (1), but I fear a general mechanism would be slower. Also, the timecounter code now extends into userland, so purely kernel locking cannot work for it. The userland part is more careful about locking than the kernel. It has memory barriers and other pessimizations which were intentionally left out of the kernel locking for timecounters. If these barriers are actually necessary, then they give the silly situation that there are less races for userland timecounting than kernel timecounting provided userland mostly does direct accesses instead of syscalls and kernel uses of timecounters are are infrequent enough to not race often with the userland accesses. (1) The main bugs are: - various calls to tc_setclock() with only Giant locking or worse. One is in settime(). settime() used to be locked by spclock(). That stopped working when SMPng removed spls and replaced them by Giant locking, and is more broken now. Giant locking is inadequate for most things related to timing. Using it in settime() gives the problem that settime() might be interrupted for many milliseconds. It then sets a time significantly in the past, assuming that it started with the correct time. Of course, it might be preempted before it aquires a sufficient lock. It never tried to recover from that. The extra breakage is that someone removed the splx() at the end, so settime() now has an splclock() with no corresponding splx(). This is only harmless since splclock() has no effect. settime()'s locking in FreeBSD-4, where it was last mostly correct, was: splclock() at start. Run almost entirely with that. But lower the spl using splsoftclock() too soon, before the end. It was apparently necessary to call lease_updatetime() at a lower priority, but the code was badly ordered so that was done before updating the time in the hardware. In fact, the hardware part was done woth no locking at all: from FreeBSD-4: % ts.tv_sec = tv->tv_sec; % ts.tv_nsec = tv->tv_usec * 1000; % set_timecounter(&ts); FreeBSD-4 had timecounters, and set_timecounter() corresponds to tc_setclock(). The locking here was provided all timecounter updates are locked by the same lock (splclock()). The most critical ones were -- timecounter updates are normally done from hardclock() and that was locked by splclock(). % (void) splsoftclock(); % lease_updatetime(delta.tv_sec); Reduce priority, but not to 0, to call lease_updatetime(). % splx(s); % resettodr(); Reduce priority to 0 to update the hardware (TODR, not timecounter). This is broken. The lock should be held throughout to increase the chance of updating the hardware to a time that is nearly current. Worse, most resettodr() implementations had and still have null locking except possibly for individual register accesses. They depend on locking from here and other callers. splclock() locking would have been adequate. % return (0); SMPng broke all this. It still doesn't even aquire Giant until near the end, leaving parts unlocked: % static int % settime(struct thread *td, struct timeval *tv) % { % struct timeval delta, tv1, tv2; % static struct timeval maxtime, laststep; % struct timespec ts; % int s; % % s = splclock(); % microtime(&tv1); No locking yet, except internally in microtime(). % delta = *tv; % timevalsub(&delta, &tv1); % % /* % * If the system is secure, we do not allow the time to be % * set to a value earlier than 1 second less than the highest % * ... % */ % if (securelevel_gt(td->td_ucred, 1) != 0) { Race with changes in the setting of securelevel. Not important to see a stale value provided we don't act nonatomically on it. % if (delta.tv_sec < 0 || delta.tv_usec < 0) { % /* % * Update maxtime to latest time we've seen. % */ % if (tv1.tv_sec > maxtime.tv_sec) % maxtime = tv1; % tv2 = *tv; % timevalsub(&tv2, &maxtime); % if (tv2.tv_sec < -1) { % tv->tv_sec = maxtime.tv_sec - 1; % printf("Time adjustment clamped to -1 second\n"); % } Race to change our state (maxtime). Giant locking for this would be plenty. % } else { % if (tv1.tv_sec == laststep.tv_sec) { % splx(s); % return (EPERM); The null spls were carefully maintained here when this return was added. % } % if (delta.tv_sec > 1) { % tv->tv_sec = tv1.tv_sec + 1; % printf("Time adjustment clamped to +1 second\n"); % } % laststep = *tv; % } % } Race to change our state (laststep). % % ts.tv_sec = tv->tv_sec; % ts.tv_nsec = tv->tv_usec * 1000; % mtx_lock(&Giant); % tc_setclock(&ts); Giant locking at last, at a point where it is almost no use. Giant locking is neither necessary or sufficient for tc_setclock(). % resettodr(); The Giant locking extends to resettodr() where it has a chance of doing some good iff all other calls to resettodr() do it too. % mtx_unlock(&Giant); % return (0); % } Someone removed lease_updatetime() and the 2 spls near it. Only one of these was related to lease_updatetime(). So splclock() is bracketed with splx() for the unusal return above, but not for the main return. - locking in kern_ntptime.c has similar nonsense involving Giant, but worse since there are callbacks from timecounter code into ntp code. This file still has a comment saying that everything must use splclock(): % * Note that all routines must run at priority splclock or higher. but it only has 1 splclock() call itself. It depends on all calls into it the be done at splclock() and for splclock() to actually work. It has a sprinkling of Giant locking. But when timecounter code calls into it, it has nothing like Giant locking. Timecounter code has some time- domain locking and perhaps still sched_lock for the usual case where it is called from the clock interrupt (settime() bypasses this). Timecounter code calls ntp code mainly for seconds update and pps. These functions provide no additional locking, but access ntptime's most critical state. Giant locking would be a gross LOR of course. The Giant locking for the syscalls isn't a LOR, but just cannot work. Another interesting comment: % #ifdef PPS_SYNC % /* % * hardpps() - discipline CPU clock oscillator to external PPS signal % * ... % * Note that, on some Unix systems this routine runs at an interrupt % * priority level higher than the timer interrupt routine hardclock(). % * Therefore, the variables used are distinct from the hardclock() % * variables, except for the actual time and frequency variables, which % * are determined by this routine and updated atomically. % */ I don't see any atomic updates. Certainly no mutexes, atomic ops or memory barriers. I think there is at best stores that are atomic on most UP systems combined with time-domain locking. The time-domain locking might even work, as in timecounter code generally, but is difficult to understand. Any variable that is only updated every second is unlikely to have a race in the update itself. Then it can be arranged that readers work if they see stale values out of order. But ntptime is mostly old UP code that depends on splclock(), so it is unlikely to arrange this. Only core timecounter code tries to arrange this. It may be too sloppy with memory barriers for the generation count, but this is unclear. Userland doesn't trust this, and sprinkles memory barriers. All userland can do to get more problems is to run stupid code that reads the timecounter in a loop. Time-domain locking obviously works better if the times between the reads are larger, and the stupid loop can hit problems sooner by making more calls. Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 04:53:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7DD6C3E4; Sat, 16 Aug 2014 04:53:01 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0204.outbound.protection.outlook.com [207.46.163.204]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D4DFA2A68; Sat, 16 Aug 2014 04:53:00 +0000 (UTC) Received: from BY2PR05CA003.namprd05.prod.outlook.com (10.242.32.33) by BY2PR05MB727.namprd05.prod.outlook.com (10.141.223.23) with Microsoft SMTP Server (TLS) id 15.0.1005.10; Sat, 16 Aug 2014 04:52:56 +0000 Received: from BY2FFO11FD009.protection.gbl (2a01:111:f400:7c0c::100) by BY2PR05CA003.outlook.office365.com (2a01:111:e400:2c2a::33) with Microsoft SMTP Server (TLS) id 15.0.1010.18 via Frontend Transport; Sat, 16 Aug 2014 04:52:56 +0000 Received: from P-EMF01-SAC.jnpr.net (66.129.239.15) by BY2FFO11FD009.mail.protection.outlook.com (10.1.14.73) with Microsoft SMTP Server (TLS) id 15.0.1010.11 via Frontend Transport; Sat, 16 Aug 2014 04:52:56 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 15 Aug 2014 21:52:56 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s7G4qsn06166; Fri, 15 Aug 2014 21:52:54 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 5F47E580A2; Fri, 15 Aug 2014 21:52:54 -0700 (PDT) To: Alfred Perlstein Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <53EEA74B.9070107@mu.org> References: <201408141640.s7EGe422096656@idle.juniper.net> <53ED57F2.5020808@mu.org> <20140815053604.9E40B580A2@chaos.jnpr.net> <53EDB0EF.6090902@mu.org> <20140815173830.93832580A2@chaos.jnpr.net> <53EEA74B.9070107@mu.org> Comments: In-reply-to: Alfred Perlstein message dated "Fri, 15 Aug 2014 17:35:23 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 15 Aug 2014 21:52:54 -0700 Message-ID: <20140816045254.5F47E580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.15; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199003)(77982001)(87936001)(50226001)(76506005)(6806004)(44976005)(85306004)(4396001)(20776003)(47776003)(89996001)(76176999)(50986999)(57986006)(83322001)(101356003)(92566001)(92726001)(93916002)(76482001)(64706001)(77156001)(46102001)(86362001)(69596002)(68736004)(107046002)(70486001)(97736001)(99396002)(80022001)(102176002)(110136001)(84676001)(81156004)(106466001)(81342001)(21056001)(31966008)(74662001)(88136002)(74502001)(79102001)(104166001)(85852003)(83072002)(90896003)(62966002)(48376002)(105596002)(81542001)(95666004)(87286001)(50466002)(102836001)(33656002)(93546004)(42262002); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB727; H:P-EMF01-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;UriScan:; X-Forefront-PRVS: 0305463112 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.15 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.15) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: Marcel Moolenaar , Phil Shafer , John-Mark Gurney , arch@freebsd.org, Poul-Henning Kamp , freebsd-arch , Konstantin Belousov X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 04:53:01 -0000 On Fri, 15 Aug 2014 17:35:23 -0700, Alfred Perlstein writes: >>> How many programs have been successfully converted over to libxo at this >>> point? >> How is that relevant to any of this discussion? >> >Well, it speaks towards the vision of getting this done in a timely >manner. As I said there is a GSOC project that has a ton of code >already done. Yes but as previously pointed out, the approach taken is far from ideal, [we previously rejected the idea of trying to contribute that approach] I think libxo will provide a much better result. >If this libxo is ready to go in, it should go in and we No objection here. There are a small number of apps that we particularly want converted, which we would propose as examples. The purpose of this thread was to illicit feedback on the idea and guage acceptance of the proposed API - which you have to admit isn't as cosy and comforting as printf, but is pretty palatable considering the functionality provided. On that front I think we are looking good. There has been very useful discussion on a number of points. I don't think I have spotted any fundamental objection to the idea. It is probably easier for Phil to commit to our internal mirror. We can take the next steps from there. >should get towards converting more utils to using it. However if we are >going to perpetually add frameworky things, but not convert over >userland tools to the actual framework, then that is a potential problem >worth calling out. Indeed. Again that's why I prefer to see this (the library at least) done by someone who's been doing this sort of thing successfuly for ages. From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 09:38:17 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CEB3F515; Sat, 16 Aug 2014 09:38:17 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C5A52568; Sat, 16 Aug 2014 09:38:17 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7G9cBvv076919 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 16 Aug 2014 12:38:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7G9cBvv076919 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7G9cBN0076918; Sat, 16 Aug 2014 12:38:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 16 Aug 2014 12:38:11 +0300 From: Konstantin Belousov To: Mateusz Guzik Subject: Re: [PATCH 1/2] Implement simple sequence counters with memory barriers. Message-ID: <20140816093811.GX2737@kib.kiev.ua> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <1408064112-573-2-git-send-email-mjguzik@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BkCcXYe/IK2urb/q" Content-Disposition: inline In-Reply-To: <1408064112-573-2-git-send-email-mjguzik@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Konstantin Belousov , Johan Schuijt , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 09:38:18 -0000 --BkCcXYe/IK2urb/q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 15, 2014 at 02:55:11AM +0200, Mateusz Guzik wrote: > --- > sys/sys/seq.h | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++ > 1 file changed, 126 insertions(+) > create mode 100644 sys/sys/seq.h >=20 > diff --git a/sys/sys/seq.h b/sys/sys/seq.h > new file mode 100644 > index 0000000..0971aef > --- /dev/null > +++ b/sys/sys/seq.h > @@ -0,0 +1,126 @@ > +/*- > + * Copyright (c) 2014 The FreeBSD Project > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distributio= n. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PU= RPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIAB= LE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUE= NTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOO= DS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, S= TRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY= WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + * > + * $FreeBSD$ > + */ > + > +#ifndef _SYS_SEQ_H_ > +#define _SYS_SEQ_H_ > + > +#ifdef _KERNEL > + > +/* > + * Typical usage: > + * > + * writers: > + * lock_exclusive(&obj->lock); > + * seq_write_begin(&obj->seq); > + * ..... > + * seq_write_end(&obj->seq); > + * unlock_exclusive(&obj->unlock); > + * > + * readers: > + * obj_t lobj; > + * seq_t seq; > + * > + * for (;;) { > + * seq =3D seq_read(&gobj->seq); > + * lobj =3D gobj; > + * if (seq_consistent(&gobj->seq, seq)) > + * break; > + * cpu_spinwait(); > + * } > + * foo(lobj); > + */ =09 > + > +typedef uint32_t seq_t; > + > +/* A hack to get MPASS macro */ > +#include > +#include > + > +#include > + > +static __inline bool > +seq_in_modify(seq_t seqp) > +{ > + > + return (seqp & 1); > +} > + > +static __inline void > +seq_write_begin(seq_t *seqp) > +{ > + > + MPASS(!seq_in_modify(*seqp)); > + (*seqp)++; > + wmb(); This probably ought to be written as atomic_add_rel_int(seqp, 1); Same note for all other linux-style barriers. In fact, on x86 wmb() is sfence and it serves no useful purpose in seq_write*. Overall, it feels too alien and linux-ish for my taste. Since we have sequence bound to some lock anyway, could we introduce some sort of generation-aware locks variants, which extend existing locks, and where lock/unlock bump generation number ? > +} > + > +static __inline void > +seq_write_end(seq_t *seqp) > +{ > + > + wmb(); > + (*seqp)++; > + MPASS(!seq_in_modify(*seqp)); > +} > + > +static __inline seq_t > +seq_read(seq_t *seqp) > +{ > + seq_t ret; > + > + for (;;) { > + ret =3D READ_ONCE(*seqp); > + if (seq_in_modify(ret)) { > + cpu_spinwait(); > + continue; > + } > + break; > + } > + > + rmb(); > + > + return (ret); > +} > + > +static __inline seq_t > +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) > +{ > + > + MPASS(!seq_in_modify(oldseqp)); > + return (*seqp =3D=3D oldseqp); > +} > + > +static __inline seq_t > +seq_consistent(seq_t *seqp, seq_t oldseqp) > +{ > + > + rmb(); > + return (seq_consistent_nomb(seqp, oldseqp)); > +} > + > +#endif /* _KERNEL */ > +#endif /* _SYS_SEQ_H_ */ > --=20 > 2.0.2 --BkCcXYe/IK2urb/q Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT7yaDAAoJEJDCuSvBvK1BTRMP/1ECsFOhc06qSXshkUgScpQ7 xbAghb6WYTI/I4Es3w9aBq6AxK1YfaxjW0Liv0/YFAkkUnSU9Ko6xpoY2QZDBkv/ 5BxCA8Qd5F02GDz7lnPt2iiCevDJXmGQqEtt3VKzE26eoinJX38v1lKvFSTMAXKq UXZ9YIsL5TrLUECoM+UuB5CtPbD7/TtQ2W6XBgqCKYGxjy9zWHysRWi8sej0kwst 3zzztnCaRGSNwDhnaKfZND21QVzmMx6budSi6Q5/UjBiFrIBXtJ+Ioq7bc+tmiOO dTUq2BFyhkN+LgqSLK/+2X8dT0qemwrJWzuAw2tcxzx6Wdq0zPAmwtZgyPar+rZh fdrbXnSHXhMaBaqZJsVyp6BRZ+TfSRNQ+wWaYdFcsme9OTMe7ciZCxaCKCYho8dX yE38GOmziBWRqu0LbL1fhQuc1XE5RmaMiUR6MEyfa/t2/RjLJJEVJ3ec+1esLB0W CdwrXeObDMMqE/IWxVIteEWOPI88r/RkNcjnLOm6aJ4JGvvqiE+ugVlsIsLZI2Qp P+Cq8+afDsQRWbwi6jZjESx1W3XecwhAcLvjLpu4KXICi8QJSCwxfMjU9pUR0weD npK1C70skOqJCi4Ty7oDHrmmUAt6NueMMIbESBfbwae/CnrQSHdii4Oc1s5X/G2+ yQmbWjUPT6CLXaLfv5fH =LoS0 -----END PGP SIGNATURE----- --BkCcXYe/IK2urb/q-- From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 18:54:12 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C799993F; Sat, 16 Aug 2014 18:54:12 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9FA2AF8; Sat, 16 Aug 2014 18:54:12 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7GIs7A4002092 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 16 Aug 2014 21:54:07 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7GIs7A4002092 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7GIs6Ra002091; Sat, 16 Aug 2014 21:54:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 16 Aug 2014 21:54:06 +0300 From: Konstantin Belousov To: Mateusz Guzik Subject: Re: [PATCH 1/2] Implement simple sequence counters with memory barriers. Message-ID: <20140816185406.GD2737@kib.kiev.ua> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <1408064112-573-2-git-send-email-mjguzik@gmail.com> <20140816093811.GX2737@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FJ766Kk/2p6HVARF" Content-Disposition: inline In-Reply-To: <20140816093811.GX2737@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Johan Schuijt , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 18:54:12 -0000 --FJ766Kk/2p6HVARF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 16, 2014 at 12:38:11PM +0300, Konstantin Belousov wrote: > On Fri, Aug 15, 2014 at 02:55:11AM +0200, Mateusz Guzik wrote: > > --- > > sys/sys/seq.h | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++ > > 1 file changed, 126 insertions(+) > > create mode 100644 sys/sys/seq.h > >=20 > > diff --git a/sys/sys/seq.h b/sys/sys/seq.h > > new file mode 100644 > > index 0000000..0971aef > > --- /dev/null > > +++ b/sys/sys/seq.h > > @@ -0,0 +1,126 @@ > > +/*- > > + * Copyright (c) 2014 The FreeBSD Project > > + * > > + * Redistribution and use in source and binary forms, with or without > > + * modification, are permitted provided that the following conditions > > + * are met: > > + * 1. Redistributions of source code must retain the above copyright > > + * notice, this list of conditions and the following disclaimer. > > + * 2. Redistributions in binary form must reproduce the above copyright > > + * notice, this list of conditions and the following disclaimer in = the > > + * documentation and/or other materials provided with the distribut= ion. > > + * > > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' = AND > > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, T= HE > > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR = PURPOSE > > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LI= ABLE > > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQ= UENTIAL > > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE G= OODS > > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTIO= N) > > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,= STRICT > > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN A= NY WAY > > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY= OF > > + * SUCH DAMAGE. > > + * > > + * $FreeBSD$ > > + */ > > + > > +#ifndef _SYS_SEQ_H_ > > +#define _SYS_SEQ_H_ > > + > > +#ifdef _KERNEL > > + > > +/* > > + * Typical usage: > > + * > > + * writers: > > + * lock_exclusive(&obj->lock); > > + * seq_write_begin(&obj->seq); > > + * ..... > > + * seq_write_end(&obj->seq); > > + * unlock_exclusive(&obj->unlock); > > + * > > + * readers: > > + * obj_t lobj; > > + * seq_t seq; > > + * > > + * for (;;) { > > + * seq =3D seq_read(&gobj->seq); > > + * lobj =3D gobj; > > + * if (seq_consistent(&gobj->seq, seq)) > > + * break; > > + * cpu_spinwait(); > > + * } > > + * foo(lobj); > > + */ =09 > > + > > +typedef uint32_t seq_t; > > + > > +/* A hack to get MPASS macro */ > > +#include > > +#include > > + > > +#include > > + > > +static __inline bool > > +seq_in_modify(seq_t seqp) > > +{ > > + > > + return (seqp & 1); > > +} > > + > > +static __inline void > > +seq_write_begin(seq_t *seqp) > > +{ > > + > > + MPASS(!seq_in_modify(*seqp)); > > + (*seqp)++; > > + wmb(); > This probably ought to be written as atomic_add_rel_int(seqp, 1); Alan Cox rightfully pointed out that better expression is v =3D *seqp + 1; = =20 atomic_store_rel_int(seqp, v); which also takes care of TSO on x86. > Same note for all other linux-style barriers. In fact, on x86 > wmb() is sfence and it serves no useful purpose in seq_write*. >=20 > Overall, it feels too alien and linux-ish for my taste. > Since we have sequence bound to some lock anyway, could we introduce > some sort of generation-aware locks variants, which extend existing > locks, and where lock/unlock bump generation number ? Still, merging it to the guts of lock implementation is right approach, IMO. >=20 > > +} > > + > > +static __inline void > > +seq_write_end(seq_t *seqp) > > +{ > > + > > + wmb(); > > + (*seqp)++; > > + MPASS(!seq_in_modify(*seqp)); > > +} > > + > > +static __inline seq_t > > +seq_read(seq_t *seqp) > > +{ > > + seq_t ret; > > + > > + for (;;) { > > + ret =3D READ_ONCE(*seqp); > > + if (seq_in_modify(ret)) { > > + cpu_spinwait(); > > + continue; > > + } > > + break; > > + } > > + > > + rmb(); > > + > > + return (ret); > > +} > > + > > +static __inline seq_t > > +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) > > +{ > > + > > + MPASS(!seq_in_modify(oldseqp)); > > + return (*seqp =3D=3D oldseqp); > > +} > > + > > +static __inline seq_t > > +seq_consistent(seq_t *seqp, seq_t oldseqp) > > +{ > > + > > + rmb(); > > + return (seq_consistent_nomb(seqp, oldseqp)); > > +} > > + > > +#endif /* _KERNEL */ > > +#endif /* _SYS_SEQ_H_ */ > > --=20 > > 2.0.2 --FJ766Kk/2p6HVARF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT76jOAAoJEJDCuSvBvK1BrQ8P/0/VURTb8c2uvicU5NNRyJdJ qFNT+pcJkWDXJ0NE02Tm8yLIqKkUcyjvOfYKz8PriXpyfA4odr+PT4SVszgNOJOV 2awtMTkZB+ci3/T/GhVm2IoqD+ulCw/cpzbpMg9rzizF7haWM8G4ie1YY59Ive6x hmFLtQDoSpq5XeNMPUsrA5T+KCEdG4WpC4O749tZfGcstPr8ajsG8u7+gLoC+9rt VMhCzRdHGyvJW+U2yO/iNLR1RepCmHMeFt+Euh9uluRVqD/LN2QO2Z9ZdA6fHYXe fhxSf2gyRoG93I4nvF7ILZ690cwoD/Y2UajTzf6D7j9VddqTPr41ka+u4HgFnQ96 W2p18Cd6SWKJbKwVY39QLApXL9iOUejP8BY1VCVc8wKk4jVqX4/+FXB+I0+B9UGM 0aUfa+GgQNQ4ZKyE+v8b46tQdftV3ebygzO+j9/fZqFkBL+llz+vVCFfLSpTzCWJ mz7v/bNmmObdHIiTTRtsiPIG6qoQTIPIENBhsPuV5Nluc3d4+QMyaD9wrMjhj5Jp oRjLBeRRJtTpdyjpeFDx1KjpoYgLzpq0e6jYGat61Sln+SJ0HhaCFCvFR4st+uMU xVcPMIwat6WxHySu8CJiv9fDPmwUpRh/bIc3AR/sOU/5rLGIPjoqIK99GRqPp8Qr ubqYG5EjczfEmPrioXsJ =/R2K -----END PGP SIGNATURE----- --FJ766Kk/2p6HVARF-- From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 19:47:17 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 171F6941 for ; Sat, 16 Aug 2014 19:47:17 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA08E2132 for ; Sat, 16 Aug 2014 19:47:16 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1XIjwS-000Gbz-UI; Sat, 16 Aug 2014 19:47:09 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s7GJl7oc045347; Sat, 16 Aug 2014 13:47:07 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+0K04+tFEl26pMCPHYeY7o X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: [PATCH 1/2] Implement simple sequence counters with memory barriers. From: Ian Lepore To: Konstantin Belousov In-Reply-To: <20140816185406.GD2737@kib.kiev.ua> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <1408064112-573-2-git-send-email-mjguzik@gmail.com> <20140816093811.GX2737@kib.kiev.ua> <20140816185406.GD2737@kib.kiev.ua> Content-Type: text/plain; charset="us-ascii" Date: Sat, 16 Aug 2014 13:47:07 -0600 Message-ID: <1408218427.56408.593.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Mateusz Guzik , Johan Schuijt , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 19:47:17 -0000 On Sat, 2014-08-16 at 21:54 +0300, Konstantin Belousov wrote: > On Sat, Aug 16, 2014 at 12:38:11PM +0300, Konstantin Belousov wrote: > > On Fri, Aug 15, 2014 at 02:55:11AM +0200, Mateusz Guzik wrote: > > > --- > > > sys/sys/seq.h | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > 1 file changed, 126 insertions(+) > > > create mode 100644 sys/sys/seq.h > > > > > > diff --git a/sys/sys/seq.h b/sys/sys/seq.h > > > new file mode 100644 > > > index 0000000..0971aef > > > --- /dev/null > > > +++ b/sys/sys/seq.h > > > @@ -0,0 +1,126 @@ > > > +/*- > > > + * Copyright (c) 2014 The FreeBSD Project > > > + * > > > + * Redistribution and use in source and binary forms, with or without > > > + * modification, are permitted provided that the following conditions > > > + * are met: > > > + * 1. Redistributions of source code must retain the above copyright > > > + * notice, this list of conditions and the following disclaimer. > > > + * 2. Redistributions in binary form must reproduce the above copyright > > > + * notice, this list of conditions and the following disclaimer in the > > > + * documentation and/or other materials provided with the distribution. > > > + * > > > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > > > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > > > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > > > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > > > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > > > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > > > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > > > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > > > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > > > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > > > + * SUCH DAMAGE. > > > + * > > > + * $FreeBSD$ > > > + */ > > > + > > > +#ifndef _SYS_SEQ_H_ > > > +#define _SYS_SEQ_H_ > > > + > > > +#ifdef _KERNEL > > > + > > > +/* > > > + * Typical usage: > > > + * > > > + * writers: > > > + * lock_exclusive(&obj->lock); > > > + * seq_write_begin(&obj->seq); > > > + * ..... > > > + * seq_write_end(&obj->seq); > > > + * unlock_exclusive(&obj->unlock); > > > + * > > > + * readers: > > > + * obj_t lobj; > > > + * seq_t seq; > > > + * > > > + * for (;;) { > > > + * seq = seq_read(&gobj->seq); > > > + * lobj = gobj; > > > + * if (seq_consistent(&gobj->seq, seq)) > > > + * break; > > > + * cpu_spinwait(); > > > + * } > > > + * foo(lobj); > > > + */ > > > + > > > +typedef uint32_t seq_t; > > > + > > > +/* A hack to get MPASS macro */ > > > +#include > > > +#include > > > + > > > +#include > > > + > > > +static __inline bool > > > +seq_in_modify(seq_t seqp) > > > +{ > > > + > > > + return (seqp & 1); > > > +} > > > + > > > +static __inline void > > > +seq_write_begin(seq_t *seqp) > > > +{ > > > + > > > + MPASS(!seq_in_modify(*seqp)); > > > + (*seqp)++; > > > + wmb(); > > This probably ought to be written as atomic_add_rel_int(seqp, 1); > Alan Cox rightfully pointed out that better expression is > v = *seqp + 1; > atomic_store_rel_int(seqp, v); > which also takes care of TSO on x86. > I'm curious why that's better than atomic_add_rel_int()? On ARM, I think the atomic add would be better than fetch/add/atomic_store. > > Same note for all other linux-style barriers. In fact, on x86 > > wmb() is sfence and it serves no useful purpose in seq_write*. > > > > Overall, it feels too alien and linux-ish for my taste. > > Since we have sequence bound to some lock anyway, could we introduce > > some sort of generation-aware locks variants, which extend existing > > locks, and where lock/unlock bump generation number ? > Still, merging it to the guts of lock implementation is right > approach, IMO. I thought the whole point of this is to avoid locks for reading and optimize the case where there is lots of concurrent reading and relatively infrequent writing. I notice that the size/duration of writing is unbounded (even by recommendation in comments) and there is no option for a reader to sleep until a write sequence is complete. It seems like that's an invitation to do bad things like wrap long (even potentially blocking) things inside some write begin/end points and leave readers spinning uselessly for a long time. The same thing could happen with spinlocks, except you know when you take a spinlock that you shouldn't be holding onto it for a long time, and you definitely know not to sleep. -- Ian > > > > > > +} > > > + > > > +static __inline void > > > +seq_write_end(seq_t *seqp) > > > +{ > > > + > > > + wmb(); > > > + (*seqp)++; > > > + MPASS(!seq_in_modify(*seqp)); > > > +} > > > + > > > +static __inline seq_t > > > +seq_read(seq_t *seqp) > > > +{ > > > + seq_t ret; > > > + > > > + for (;;) { > > > + ret = READ_ONCE(*seqp); > > > + if (seq_in_modify(ret)) { > > > + cpu_spinwait(); > > > + continue; > > > + } > > > + break; > > > + } > > > + > > > + rmb(); > > > + > > > + return (ret); > > > +} > > > + > > > +static __inline seq_t > > > +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) > > > +{ > > > + > > > + MPASS(!seq_in_modify(oldseqp)); > > > + return (*seqp == oldseqp); > > > +} > > > + > > > +static __inline seq_t > > > +seq_consistent(seq_t *seqp, seq_t oldseqp) > > > +{ > > > + > > > + rmb(); > > > + return (seq_consistent_nomb(seqp, oldseqp)); > > > +} > > > + > > > +#endif /* _KERNEL */ > > > +#endif /* _SYS_SEQ_H_ */ > > > -- > > > 2.0.2 > > From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 21:16:44 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5EFAC322; Sat, 16 Aug 2014 21:16:44 +0000 (UTC) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F56F2ADB; Sat, 16 Aug 2014 21:16:43 +0000 (UTC) Received: by mail-ig0-f178.google.com with SMTP id uq10so4679222igb.5 for ; Sat, 16 Aug 2014 14:16:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=u9fnfJ6Ns8ITbZ9hEEiiTXr7iyxGRuq5lbbvsjTHGNU=; b=IUfw6YPWki4HAJQj5C3iA6IOYFFkB0WVSvX6EMR4ZCKn5m1ZlKLP1d85hjl+4ULByy 6hJIzAnns0O2oSZ2vkP7EHT96o1StP+Ex09G5lpiHUKifLxAweJAJ8DombHQhEb8L1WE KFejiYDs3pIME2Frm0sy7KaA+3IO0mBCxCn02WQuurFWLqlMp4sAVOBdeoEx1OAyMzyC wp+8GsmhkaeBQ3IkDGtARsaM3bhPxO0QvLY1Ba/3kmhl+U7JLhszNcbJ8mWai4iti6HN Gq61hAfasCzvlK7h3Syfq7GL4nuYHszUEIj/xp+a6EHExvEyp/rLNVaG3CiVkPKbRDhj LupQ== MIME-Version: 1.0 X-Received: by 10.43.129.74 with SMTP id hh10mr27089276icc.48.1408223803327; Sat, 16 Aug 2014 14:16:43 -0700 (PDT) Received: by 10.43.17.196 with HTTP; Sat, 16 Aug 2014 14:16:43 -0700 (PDT) Reply-To: alc@freebsd.org In-Reply-To: <1408218427.56408.593.camel@revolution.hippie.lan> References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <1408064112-573-2-git-send-email-mjguzik@gmail.com> <20140816093811.GX2737@kib.kiev.ua> <20140816185406.GD2737@kib.kiev.ua> <1408218427.56408.593.camel@revolution.hippie.lan> Date: Sat, 16 Aug 2014 16:16:43 -0500 Message-ID: Subject: Re: [PATCH 1/2] Implement simple sequence counters with memory barriers. From: Alan Cox To: Ian Lepore Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Konstantin Belousov , Mateusz Guzik , Johan Schuijt , FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 21:16:44 -0000 On Sat, Aug 16, 2014 at 2:47 PM, Ian Lepore wrote: > On Sat, 2014-08-16 at 21:54 +0300, Konstantin Belousov wrote: > > On Sat, Aug 16, 2014 at 12:38:11PM +0300, Konstantin Belousov wrote: > > > On Fri, Aug 15, 2014 at 02:55:11AM +0200, Mateusz Guzik wrote: > > > > --- > > > > sys/sys/seq.h | 126 > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > 1 file changed, 126 insertions(+) > > > > create mode 100644 sys/sys/seq.h > > > > > > > > diff --git a/sys/sys/seq.h b/sys/sys/seq.h > > > > new file mode 100644 > > > > index 0000000..0971aef > > > > --- /dev/null > > > > +++ b/sys/sys/seq.h > > > > @@ -0,0 +1,126 @@ > > > > +/*- > > > > + * Copyright (c) 2014 The FreeBSD Project > > > > + * > > > > + * Redistribution and use in source and binary forms, with or > without > > > > + * modification, are permitted provided that the following > conditions > > > > + * are met: > > > > + * 1. Redistributions of source code must retain the above copyright > > > > + * notice, this list of conditions and the following disclaimer. > > > > + * 2. Redistributions in binary form must reproduce the above > copyright > > > > + * notice, this list of conditions and the following disclaimer > in the > > > > + * documentation and/or other materials provided with the > distribution. > > > > + * > > > > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS > IS'' AND > > > > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > TO, THE > > > > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A > PARTICULAR PURPOSE > > > > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE > LIABLE > > > > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > CONSEQUENTIAL > > > > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > SUBSTITUTE GOODS > > > > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS > INTERRUPTION) > > > > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > CONTRACT, STRICT > > > > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING > IN ANY WAY > > > > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE > POSSIBILITY OF > > > > + * SUCH DAMAGE. > > > > + * > > > > + * $FreeBSD$ > > > > + */ > > > > + > > > > +#ifndef _SYS_SEQ_H_ > > > > +#define _SYS_SEQ_H_ > > > > + > > > > +#ifdef _KERNEL > > > > + > > > > +/* > > > > + * Typical usage: > > > > + * > > > > + * writers: > > > > + * lock_exclusive(&obj->lock); > > > > + * seq_write_begin(&obj->seq); > > > > + * ..... > > > > + * seq_write_end(&obj->seq); > > > > + * unlock_exclusive(&obj->unlock); > > > > + * > > > > + * readers: > > > > + * obj_t lobj; > > > > + * seq_t seq; > > > > + * > > > > + * for (;;) { > > > > + * seq = seq_read(&gobj->seq); > > > > + * lobj = gobj; > > > > + * if (seq_consistent(&gobj->seq, seq)) > > > > + * break; > > > > + * cpu_spinwait(); > > > > + * } > > > > + * foo(lobj); > > > > + */ > > > > + > > > > +typedef uint32_t seq_t; > > > > + > > > > +/* A hack to get MPASS macro */ > > > > +#include > > > > +#include > > > > + > > > > +#include > > > > + > > > > +static __inline bool > > > > +seq_in_modify(seq_t seqp) > > > > +{ > > > > + > > > > + return (seqp & 1); > > > > +} > > > > + > > > > +static __inline void > > > > +seq_write_begin(seq_t *seqp) > > > > +{ > > > > + > > > > + MPASS(!seq_in_modify(*seqp)); > > > > + (*seqp)++; > > > > + wmb(); > > > This probably ought to be written as atomic_add_rel_int(seqp, 1); > > Alan Cox rightfully pointed out that better expression is > > v = *seqp + 1; > > atomic_store_rel_int(seqp, v); > > which also takes care of TSO on x86. > > > > I'm curious why that's better than atomic_add_rel_int()? On ARM, I > think the atomic add would be better than fetch/add/atomic_store. > > That seems unlikely. atomic_add_rel_int() has to do two things: (1) a load-linked/store-conditional loop to perform the atomic add and (2) a memory barrier to implement the release semantics. In this case, he doesn't need the add to be atomic; he only needs the release semantics. > > > Same note for all other linux-style barriers. In fact, on x86 > > > wmb() is sfence and it serves no useful purpose in seq_write*. > > > > > > Overall, it feels too alien and linux-ish for my taste. > > > Since we have sequence bound to some lock anyway, could we introduce > > > some sort of generation-aware locks variants, which extend existing > > > locks, and where lock/unlock bump generation number ? > > Still, merging it to the guts of lock implementation is right > > approach, IMO. > > I thought the whole point of this is to avoid locks for reading and > optimize the case where there is lots of concurrent reading and > relatively infrequent writing. > > I notice that the size/duration of writing is unbounded (even by > recommendation in comments) and there is no option for a reader to sleep > until a write sequence is complete. It seems like that's an invitation > to do bad things like wrap long (even potentially blocking) things > inside some write begin/end points and leave readers spinning uselessly > for a long time. The same thing could happen with spinlocks, except you > know when you take a spinlock that you shouldn't be holding onto it for > a long time, and you definitely know not to sleep. > > -- Ian > > > > > > > > > > +} > > > > + > > > > +static __inline void > > > > +seq_write_end(seq_t *seqp) > > > > +{ > > > > + > > > > + wmb(); > > > > + (*seqp)++; > > > > + MPASS(!seq_in_modify(*seqp)); > > > > +} > > > > + > > > > +static __inline seq_t > > > > +seq_read(seq_t *seqp) > > > > +{ > > > > + seq_t ret; > > > > + > > > > + for (;;) { > > > > + ret = READ_ONCE(*seqp); > > > > + if (seq_in_modify(ret)) { > > > > + cpu_spinwait(); > > > > + continue; > > > > + } > > > > + break; > > > > + } > > > > + > > > > + rmb(); > > > > + > > > > + return (ret); > > > > +} > > > > + > > > > +static __inline seq_t > > > > +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) > > > > +{ > > > > + > > > > + MPASS(!seq_in_modify(oldseqp)); > > > > + return (*seqp == oldseqp); > > > > +} > > > > + > > > > +static __inline seq_t > > > > +seq_consistent(seq_t *seqp, seq_t oldseqp) > > > > +{ > > > > + > > > > + rmb(); > > > > + return (seq_consistent_nomb(seqp, oldseqp)); > > > > +} > > > > + > > > > +#endif /* _KERNEL */ > > > > +#endif /* _SYS_SEQ_H_ */ > > > > -- > > > > 2.0.2 > > > > > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Sat Aug 16 21:47:46 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91D0A232; Sat, 16 Aug 2014 21:47:46 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F08E2040; Sat, 16 Aug 2014 21:47:45 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1XIlpA-000Mcz-BT; Sat, 16 Aug 2014 21:47:44 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s7GLlgss045559; Sat, 16 Aug 2014 15:47:42 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+wzOBUuwiyQpxhA0wS3vEp X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: [PATCH 1/2] Implement simple sequence counters with memory barriers. From: Ian Lepore To: alc@freebsd.org In-Reply-To: References: <1408064112-573-1-git-send-email-mjguzik@gmail.com> <1408064112-573-2-git-send-email-mjguzik@gmail.com> <20140816093811.GX2737@kib.kiev.ua> <20140816185406.GD2737@kib.kiev.ua> <1408218427.56408.593.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sat, 16 Aug 2014 15:47:41 -0600 Message-ID: <1408225661.56408.598.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , Mateusz Guzik , Johan Schuijt , FreeBSD Arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2014 21:47:46 -0000 On Sat, 2014-08-16 at 16:16 -0500, Alan Cox wrote: > On Sat, Aug 16, 2014 at 2:47 PM, Ian Lepore wrote: > > > On Sat, 2014-08-16 at 21:54 +0300, Konstantin Belousov wrote: > > > On Sat, Aug 16, 2014 at 12:38:11PM +0300, Konstantin Belousov wrote: > > > > On Fri, Aug 15, 2014 at 02:55:11AM +0200, Mateusz Guzik wrote: > > > > > --- > > > > > sys/sys/seq.h | 126 > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > 1 file changed, 126 insertions(+) > > > > > create mode 100644 sys/sys/seq.h > > > > > > > > > > diff --git a/sys/sys/seq.h b/sys/sys/seq.h > > > > > new file mode 100644 > > > > > index 0000000..0971aef > > > > > --- /dev/null > > > > > +++ b/sys/sys/seq.h > > > > > @@ -0,0 +1,126 @@ > > > > > +/*- > > > > > + * Copyright (c) 2014 The FreeBSD Project > > > > > + * > > > > > + * Redistribution and use in source and binary forms, with or > > without > > > > > + * modification, are permitted provided that the following > > conditions > > > > > + * are met: > > > > > + * 1. Redistributions of source code must retain the above copyright > > > > > + * notice, this list of conditions and the following disclaimer. > > > > > + * 2. Redistributions in binary form must reproduce the above > > copyright > > > > > + * notice, this list of conditions and the following disclaimer > > in the > > > > > + * documentation and/or other materials provided with the > > distribution. > > > > > + * > > > > > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS > > IS'' AND > > > > > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > > TO, THE > > > > > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A > > PARTICULAR PURPOSE > > > > > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE > > LIABLE > > > > > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > > CONSEQUENTIAL > > > > > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > > SUBSTITUTE GOODS > > > > > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS > > INTERRUPTION) > > > > > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > > CONTRACT, STRICT > > > > > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING > > IN ANY WAY > > > > > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE > > POSSIBILITY OF > > > > > + * SUCH DAMAGE. > > > > > + * > > > > > + * $FreeBSD$ > > > > > + */ > > > > > + > > > > > +#ifndef _SYS_SEQ_H_ > > > > > +#define _SYS_SEQ_H_ > > > > > + > > > > > +#ifdef _KERNEL > > > > > + > > > > > +/* > > > > > + * Typical usage: > > > > > + * > > > > > + * writers: > > > > > + * lock_exclusive(&obj->lock); > > > > > + * seq_write_begin(&obj->seq); > > > > > + * ..... > > > > > + * seq_write_end(&obj->seq); > > > > > + * unlock_exclusive(&obj->unlock); > > > > > + * > > > > > + * readers: > > > > > + * obj_t lobj; > > > > > + * seq_t seq; > > > > > + * > > > > > + * for (;;) { > > > > > + * seq = seq_read(&gobj->seq); > > > > > + * lobj = gobj; > > > > > + * if (seq_consistent(&gobj->seq, seq)) > > > > > + * break; > > > > > + * cpu_spinwait(); > > > > > + * } > > > > > + * foo(lobj); > > > > > + */ > > > > > + > > > > > +typedef uint32_t seq_t; > > > > > + > > > > > +/* A hack to get MPASS macro */ > > > > > +#include > > > > > +#include > > > > > + > > > > > +#include > > > > > + > > > > > +static __inline bool > > > > > +seq_in_modify(seq_t seqp) > > > > > +{ > > > > > + > > > > > + return (seqp & 1); > > > > > +} > > > > > + > > > > > +static __inline void > > > > > +seq_write_begin(seq_t *seqp) > > > > > +{ > > > > > + > > > > > + MPASS(!seq_in_modify(*seqp)); > > > > > + (*seqp)++; > > > > > + wmb(); > > > > This probably ought to be written as atomic_add_rel_int(seqp, 1); > > > Alan Cox rightfully pointed out that better expression is > > > v = *seqp + 1; > > > atomic_store_rel_int(seqp, v); > > > which also takes care of TSO on x86. > > > > > > > I'm curious why that's better than atomic_add_rel_int()? On ARM, I > > think the atomic add would be better than fetch/add/atomic_store. > > > > > > That seems unlikely. atomic_add_rel_int() has to do two things: (1) a > load-linked/store-conditional loop to perform the atomic add and (2) a > memory barrier to implement the release semantics. In this case, he > doesn't need the add to be atomic; he only needs the release semantics. > For some reason I was thinking the two operations were essentially identical on armv6 except for an add instruction in the loop. That turns out to be the case for 64-bit values, but not for 32; my bad. -- Ian > > > > > > Same note for all other linux-style barriers. In fact, on x86 > > > > wmb() is sfence and it serves no useful purpose in seq_write*. > > > > > > > > Overall, it feels too alien and linux-ish for my taste. > > > > Since we have sequence bound to some lock anyway, could we introduce > > > > some sort of generation-aware locks variants, which extend existing > > > > locks, and where lock/unlock bump generation number ? > > > Still, merging it to the guts of lock implementation is right > > > approach, IMO. > > > > I thought the whole point of this is to avoid locks for reading and > > optimize the case where there is lots of concurrent reading and > > relatively infrequent writing. > > > > I notice that the size/duration of writing is unbounded (even by > > recommendation in comments) and there is no option for a reader to sleep > > until a write sequence is complete. It seems like that's an invitation > > to do bad things like wrap long (even potentially blocking) things > > inside some write begin/end points and leave readers spinning uselessly > > for a long time. The same thing could happen with spinlocks, except you > > know when you take a spinlock that you shouldn't be holding onto it for > > a long time, and you definitely know not to sleep. > > > > -- Ian > > > > > > > > > > > > > > +} > > > > > + > > > > > +static __inline void > > > > > +seq_write_end(seq_t *seqp) > > > > > +{ > > > > > + > > > > > + wmb(); > > > > > + (*seqp)++; > > > > > + MPASS(!seq_in_modify(*seqp)); > > > > > +} > > > > > + > > > > > +static __inline seq_t > > > > > +seq_read(seq_t *seqp) > > > > > +{ > > > > > + seq_t ret; > > > > > + > > > > > + for (;;) { > > > > > + ret = READ_ONCE(*seqp); > > > > > + if (seq_in_modify(ret)) { > > > > > + cpu_spinwait(); > > > > > + continue; > > > > > + } > > > > > + break; > > > > > + } > > > > > + > > > > > + rmb(); > > > > > + > > > > > + return (ret); > > > > > +} > > > > > + > > > > > +static __inline seq_t > > > > > +seq_consistent_nomb(seq_t *seqp, seq_t oldseqp) > > > > > +{ > > > > > + > > > > > + MPASS(!seq_in_modify(oldseqp)); > > > > > + return (*seqp == oldseqp); > > > > > +} > > > > > + > > > > > +static __inline seq_t > > > > > +seq_consistent(seq_t *seqp, seq_t oldseqp) > > > > > +{ > > > > > + > > > > > + rmb(); > > > > > + return (seq_consistent_nomb(seqp, oldseqp)); > > > > > +} > > > > > + > > > > > +#endif /* _KERNEL */ > > > > > +#endif /* _SYS_SEQ_H_ */ > > > > > -- > > > > > 2.0.2 > > > > > > > > > > > > _______________________________________________ > > freebsd-arch@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"