From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 11:41:12 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7861951F; Sun, 13 Jan 2013 11:41:12 +0000 (UTC) (envelope-from edschouten@gmail.com) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) by mx1.freebsd.org (Postfix) with ESMTP id 87873F9; Sun, 13 Jan 2013 11:41:11 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id l5so2307642lbo.37 for ; Sun, 13 Jan 2013 03:41:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4jKli5m4EB7eIUJUiMO91zQDfEmM6vAQ/RCk2bixb3Q=; b=SWOUqwqaGMBiHDQss0QRu49nudnw45KceGSRdOE+YizXNv/l60z1euYZqKK8B5WnXz 55Umf74QqhC5fNpM8WOVvpbCNXIk+D+3/kO1apO0OS7L+M2y8mgh3iM6oXCFkbuAmWSn CkpiNgbpSGalXFqmRui3lGPH4pB6GGc7xlFQknsI/03We1uJYC6WJVtIARJkIo2qR1TN RLeTBEG7sFlL3TUX94+w0zH+G7LEnsDA1wyBpQaa2YelG2XzgbNKxDuHDtOMn7SMFoHa 3rYSsSF1yugfEVchiCDdo3kfv98tBWRs/cLPL37LUdStN1OnK+wKoyB1+QE8CHex3T0K PBAg== MIME-Version: 1.0 Received: by 10.112.29.104 with SMTP id j8mr33916166lbh.0.1358077269998; Sun, 13 Jan 2013 03:41:09 -0800 (PST) Sender: edschouten@gmail.com Received: by 10.112.86.6 with HTTP; Sun, 13 Jan 2013 03:41:09 -0800 (PST) In-Reply-To: <20130107172433.GX82219@kib.kiev.ua> References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> Date: Sun, 13 Jan 2013 12:41:09 +0100 X-Google-Sender-Auth: BjCr3hUA_5zzHOmzFatHQ1aJy2Y Message-ID: Subject: Re: LLVM Image Activator From: Ed Schouten To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: freebsd-toolchain@freebsd.org, John Baldwin , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 11:41:12 -0000 Hi Kostik, 2013/1/7 Konstantin Belousov : > I still do remember the buzz about the binary format 0xCAFEBABE, which > AFAIR gained image activator support on several OSes, to be garbage > collected. Maybe it would then be a good idea then to add some kind of general purpose remapping imgact? Example: /etc/imgacttab: cafebabe /usr/local/bin/java cffaedfe /usr/local/bin/osx_emulator 4243c0de /usr/bin/lli That way we still give people the freedom to play around with mapping their own executable formats, but don't need to maintain a bunch of imgacts. -- Ed Schouten From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 13:21:12 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 73F81617; Sun, 13 Jan 2013 13:21:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id E35703F5; Sun, 13 Jan 2013 13:21:11 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DDKvSg039763; Sun, 13 Jan 2013 15:20:57 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DDKvSg039763 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DDKvOh039762; Sun, 13 Jan 2013 15:20:57 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 15:20:57 +0200 From: Konstantin Belousov To: Ed Schouten Subject: Re: LLVM Image Activator Message-ID: <20130113132057.GQ2561@kib.kiev.ua> References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="htVlOAOs/VIaEhiC" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-toolchain@freebsd.org, John Baldwin , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 13:21:12 -0000 --htVlOAOs/VIaEhiC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: > Hi Kostik, >=20 > 2013/1/7 Konstantin Belousov : > > I still do remember the buzz about the binary format 0xCAFEBABE, which > > AFAIR gained image activator support on several OSes, to be garbage > > collected. >=20 > Maybe it would then be a good idea then to add some kind of general > purpose remapping imgact? Example: >=20 > /etc/imgacttab: >=20 > cafebabe /usr/local/bin/java > cffaedfe /usr/local/bin/osx_emulator > 4243c0de /usr/bin/lli >=20 > That way we still give people the freedom to play around with mapping > their own executable formats, but don't need to maintain a bunch of > imgacts. A generic module that could be somewhat customized at runtime to map offset+signature into the shebang path could be a possibility indeed. I strongly prefer to have it as module and not enabled by default. Asking Nathan for writing the thing is too much, IMHO, esp. in the response to the 50-lines hack. --htVlOAOs/VIaEhiC Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8rS5AAoJEJDCuSvBvK1BOS0QAIFUrY+SSfw7mrzHFFIlkdZr V1f/BD6ttZPTLWjxzkFh99xOgYikcxdwBHriz+y15pjNfp4qdsq1ShUvkeslGgfj mjmXXQeLcSERybNokzFzRJ4PehAQ7Nb0GjkYFZVstPBcBIF1WPgYBfw2m+LAXbwZ ru9WC4shXYyqX7i7vv3eJ5vybJAiy0KUSs1q24H6Vmxc3ouIBY9WXYtWO7HNw0oC SYuQ2Ns43qK36c3SpKFUSrAoFgyuVLR/aKfCCo0I3qeV0Wn+wmhE9Feni5I+q2Ar Qgjz23h3ZHTiux8bLa1CdWXt2Ayqa1hDzVTthPnsCJHLrLwbc7VcwCsW9i2O90ts Dm9irz/w9vneG7HIYgw2YAQKq9AJHSjLy0+OVXUpqnrO4LzfW1VHNh3h3tqlUJJ1 qhlOfzYb8qv0w/5Y2vMbP+fWFH76aFJeVCGSDd+bHSUvVqdICUNZFJRzlW3aAi09 xaZiF63Oi7kaU2gydCZ/pimVCPBaKsqlzkfwY++V0z6sVincxnnO2NqKyv3gYsXQ 4ARcChMpOJHqAzt9MlxQXkXIgE380rxnMWMB6VC66graZJ9Kx1QglZA1AN809w1e tZ2R5PM/ySwM4fYpw8VxHXhegEsG1aIirlVIWZQ/P6pcOgmp43ZF7SOk1Yi5MRQA igJsuK5VjMZpijPmo6mO =4znq -----END PGP SIGNATURE----- --htVlOAOs/VIaEhiC-- From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 13:32:02 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D8D0A999; Sun, 13 Jan 2013 13:32:02 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id 9E2C963E; Sun, 13 Jan 2013 13:32:02 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 386A5359315; Sun, 13 Jan 2013 14:32:00 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 197C72848C; Sun, 13 Jan 2013 14:32:00 +0100 (CET) Date: Sun, 13 Jan 2013 14:32:00 +0100 From: Jilles Tjoelker To: Konstantin Belousov , davidxu@freebsd.org Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130113133159.GA72462@stack.nl> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130112162547.GA54954@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 13:32:02 -0000 On Sat, Jan 12, 2013 at 05:25:47PM +0100, Jilles Tjoelker wrote: > This suggests a different rather simpler change. Libthr can always use > its rtld lock implementation instead of only when multiple threads > exist. This avoids the sigprocmask() syscalls and should not be much > slower than the default implementation in rtld because that also > contains atomic operations (both the unpatched and the patched version). > People that care about performance of exceptions can then link in libthr > (in many cases, it is already linked in to allow for (the possibility > of) threading). > I have tested this and exceptions were indeed more than twice as fast in > my test program if I created an extra thread doing nothing than in the > fully single-threaded version (with or without libthr). Here is a patch. It is lightly tested. The function _thr_rtld_fini() can be removed afterwards because it is no longer used. Index: lib/libthr/thread/thr_init.c =================================================================== --- lib/libthr/thread/thr_init.c (revision 244639) +++ lib/libthr/thread/thr_init.c (working copy) @@ -363,6 +363,7 @@ _thr_signal_init(); if (_thread_event_mask & TD_CREATE) _thr_report_creation(curthread, curthread); + _thr_rtld_init(); } } Index: lib/libthr/thread/thr_kern.c =================================================================== --- lib/libthr/thread/thr_kern.c (revision 244639) +++ lib/libthr/thread/thr_kern.c (working copy) @@ -57,11 +57,6 @@ return (0); __isthreaded = threaded; - if (threaded != 0) { - _thr_rtld_init(); - } else { - _thr_rtld_fini(); - } return (0); } -- Jilles Tjoelker From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 13:51:17 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5DEF9B4A; Sun, 13 Jan 2013 13:51:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id AC6306A7; Sun, 13 Jan 2013 13:51:16 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DDpAYC042906; Sun, 13 Jan 2013 15:51:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DDpAYC042906 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DDpAKQ042905; Sun, 13 Jan 2013 15:51:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 15:51:10 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130113135110.GS2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> <20130113133159.GA72462@stack.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uc+LCsvbRhlIZa01" Content-Disposition: inline In-Reply-To: <20130113133159.GA72462@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 13:51:17 -0000 --uc+LCsvbRhlIZa01 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 02:32:00PM +0100, Jilles Tjoelker wrote: > On Sat, Jan 12, 2013 at 05:25:47PM +0100, Jilles Tjoelker wrote: > > This suggests a different rather simpler change. Libthr can always use > > its rtld lock implementation instead of only when multiple threads > > exist. This avoids the sigprocmask() syscalls and should not be much > > slower than the default implementation in rtld because that also > > contains atomic operations (both the unpatched and the patched version). > > People that care about performance of exceptions can then link in libthr > > (in many cases, it is already linked in to allow for (the possibility > > of) threading). >=20 > > I have tested this and exceptions were indeed more than twice as fast in > > my test program if I created an extra thread doing nothing than in the > > fully single-threaded version (with or without libthr). >=20 > Here is a patch. It is lightly tested. >=20 > The function _thr_rtld_fini() can be removed afterwards because it is no > longer used. >=20 > Index: lib/libthr/thread/thr_init.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- lib/libthr/thread/thr_init.c (revision 244639) > +++ lib/libthr/thread/thr_init.c (working copy) > @@ -363,6 +363,7 @@ > _thr_signal_init(); > if (_thread_event_mask & TD_CREATE) > _thr_report_creation(curthread, curthread); > + _thr_rtld_init(); Please add a comment there, describing the reason for initializing threaded rtld locks. > } > } > =20 > Index: lib/libthr/thread/thr_kern.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- lib/libthr/thread/thr_kern.c (revision 244639) > +++ lib/libthr/thread/thr_kern.c (working copy) > @@ -57,11 +57,6 @@ > return (0); > =20 > __isthreaded =3D threaded; > - if (threaded !=3D 0) { > - _thr_rtld_init(); > - } else { > - _thr_rtld_fini(); > - } > return (0); > } I do not object. --uc+LCsvbRhlIZa01 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8rvNAAoJEJDCuSvBvK1B2cgP/3syRxWJGr+UxICBCFjzuzTf qNJAC8zs4/ZaGLOaH3iRlQ13OnaR5JPq2PTIr5tImuy9f5/ixW0APcSfVq0fBUkF 68ZmPS7ACXvbclkJKVDJE0vazD+vrQ48Tl2KzeHX7ELVYaJI2yzAr7l/wF+z+Dmh GQu/rTS0J/uxjdSXnBwo53oIM2+boab+333Gvz5eUAY1xabPZBnlCoHqf6lhtFcy bIxVmQYRLix3kbRAyXCOtQvCOfKDSFyV7bi8E5UUSPjigXWSjCWXlzKaKT5xsbK1 tOyKFPObOte1GYkhj9nXe+fukImAsgUUeJlP6+42xGPajMhM41IVog8U6wuIjOB9 6VaXc8U1tr+r9zL4uWD/MW7cIThv/VL3Id8N+A1roIxvB/axJrMx7noGA/r/+89k f6zm4BxymXv100tZ7u7bT8VIDjkp4fpQXZCRdC9Qoqvi7UcfhJ7gxKUbbbOQf5Uo 893ysPRxW7oQpSgYiFyrrYTFL1e5mCEqVzO8PKuEv6BS5tuuPdeK1uRlRKTKrfW0 V1lBTFv7JehlLW6gYqmFyokDdbO/6zQmnUMeQIjtYfX+0xIq8UgiODBn8TsqzqlL F6afbDI+d8/2RrSm0vajSzYYPRoXjkbRwfLHEQ3Lbm1oiH7ifnKv3/3pNUMVACDW 0BKhbXfXmfIK78knovi5 =006m -----END PGP SIGNATURE----- --uc+LCsvbRhlIZa01-- From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 14:52:51 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B92D6A67; Sun, 13 Jan 2013 14:52:51 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 67D9788D; Sun, 13 Jan 2013 14:52:51 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id r0DEoHV8048391; Sun, 13 Jan 2013 07:50:24 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r0DEoEEO000770; Sun, 13 Jan 2013 07:50:14 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: Fast sigblock (AKA rtld speedup) From: Ian Lepore To: John-Mark Gurney In-Reply-To: <20130112230435.GJ1410@funkthat.com> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130112230435.GJ1410@funkthat.com> Content-Type: text/plain; charset="us-ascii" Date: Sun, 13 Jan 2013 07:50:14 -0700 Message-ID: <1358088614.32417.23.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 14:52:51 -0000 On Sat, 2013-01-12 at 15:04 -0800, John-Mark Gurney wrote: > David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > > and can not be freed until process is exited, the page is doubly > > mapped into in kernel and userland, accessing the shared data > > in kernel has zero overhead though. > > Don't forget that there are arches out there w/ VIVT caches which will > probably eliminate most of the performance benifits if we have the same > page mapped writable in two different virtual addresses.. > Even worse than eliminate the benefits, since multiple mappings with one writable disables caching on the whole page, there can be a big penalty depending on what other data is nearby that suddenly becomes uncacheable. I was initially very interested in the work to read system clocks without a syscall until I realized it was going to suffer from the same problem. -- Ian From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 15:06:47 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 709DECAA; Sun, 13 Jan 2013 15:06:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id F1C73909; Sun, 13 Jan 2013 15:06:46 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DF6XjZ050925; Sun, 13 Jan 2013 17:06:33 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DF6XjZ050925 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DF6Xih050924; Sun, 13 Jan 2013 17:06:33 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 17:06:33 +0200 From: Konstantin Belousov To: Ian Lepore Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130113150633.GV2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130112230435.GJ1410@funkthat.com> <1358088614.32417.23.camel@revolution.hippie.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xxgoS+Xwt19NyhFM" Content-Disposition: inline In-Reply-To: <1358088614.32417.23.camel@revolution.hippie.lan> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, John-Mark Gurney , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 15:06:47 -0000 --xxgoS+Xwt19NyhFM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 07:50:14AM -0700, Ian Lepore wrote: > On Sat, 2013-01-12 at 15:04 -0800, John-Mark Gurney wrote: > > David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > > > and can not be freed until process is exited, the page is doubly > > > mapped into in kernel and userland, accessing the shared data > > > in kernel has zero overhead though. > >=20 > > Don't forget that there are arches out there w/ VIVT caches which will > > probably eliminate most of the performance benifits if we have the same > > page mapped writable in two different virtual addresses.. > >=20 >=20 > Even worse than eliminate the benefits, since multiple mappings with one > writable disables caching on the whole page, there can be a big penalty > depending on what other data is nearby that suddenly becomes > uncacheable. I was initially very interested in the work to read system > clocks without a syscall until I realized it was going to suffer from > the same problem. Since only kernel writes to the shared page, it should be not a problem. At least there is a specific point where you can insert the neccessary cache flush commands. Also, I suspect that even for arms, the writes are executed from the same core, which could simplify things even more. --xxgoS+Xwt19NyhFM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8s14AAoJEJDCuSvBvK1BgsAP/3I1KlKAMW9MESzj+WJBnKXh 5mhjDWuogY6QN9Cyg03nOcNO5OBLsXBSDB/rWT6qy8zyWqcMpPQ5QJ2y6gktn24L 09Ex6lTUZ8Ej9zIVsLhvdBTUSasr4+GqOEWHFT/iR2WG0PJqmDUwAEOQGHuQ7tq/ 6bmFzlZvFq5UjiBpGOXrr8Ip1j7umbKdYQ5GwZqz2OAxFsQbXEYLPRBhuK5a5mfU Tmtn2iYTethep5ey5cgUj2DfzKxM6cM/YNow0bFefoMPwXZ4taGx1xv+wfcgEs/K RINT3ER7y0t0UtJQFZGBNmX+tTdGCC8v0nHIJI6s98iQu+Pb/Q87E3bXZ+KIoQsf jkP0ENSAIUOmeLs55pt/5REe/e0hXadLr8D43qHUrvCDdKHfJBTKo2ZMSutlDA9g G+9r7PVMIw9egY3V55/l1jZB3szqc8MG6d2I4WgNh3NFXlsL83RhnVwiBhcbXi8k Wn4RLE5/9TpfoBRP4nSOkPEh+rEqVu4GIhsFJSXZ7usiJlxcgnWGSCuMj3M31UKe Vixvqi7+jaHVAzjqpPythaxW/19e2HxX7ZE5ABAMVSqAjYC7BNbf0y7Nr+i765Qj 1wtz7UX19y38GNaHjVbAFYs2Hhm79rjHTqhlZ39u8XSFgTU1MFLPpMXMOgUdCQ7j iRKhKtEVaWRIjPREA7q9 =BbUz -----END PGP SIGNATURE----- --xxgoS+Xwt19NyhFM-- From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 16:21:40 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 65F8FAC7; Sun, 13 Jan 2013 16:21:40 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from agogare.doit.wisc.edu (agogare.doit.wisc.edu [144.92.197.211]) by mx1.freebsd.org (Postfix) with ESMTP id 3FF39D80; Sun, 13 Jan 2013 16:21:40 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) id <0MGK00300O432900@smtpauth2.wiscmail.wisc.edu>; Sun, 13 Jan 2013 10:21:39 -0600 (CST) Received: from wanderer.tachypleus.net (c-24-63-204-107.hsd1.ct.comcast.net [24.63.204.107]) by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) with ESMTPSA id <0MGK000KJO413W10@smtpauth2.wiscmail.wisc.edu>; Sun, 13 Jan 2013 10:21:38 -0600 (CST) Date: Sun, 13 Jan 2013 08:21:37 -0800 From: Nathan Whitehorn Subject: Re: LLVM Image Activator In-reply-to: <20130113132057.GQ2561@kib.kiev.ua> Sender: whitehorn@wisc.edu To: Konstantin Belousov Message-id: <50F2DF11.50202@freebsd.org> X-Spam-Report: AuthenticatedSender=yes, SenderIP=24.63.204.107 X-Spam-PmxInfo: Server=avs-15, Version=5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2013.1.13.161217, SenderIP=24.63.204.107 References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> <20130113132057.GQ2561@kib.kiev.ua> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130112 Thunderbird/17.0.2 Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 16:21:40 -0000 On 01/13/13 05:20, Konstantin Belousov wrote: > On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: >> Hi Kostik, >> >> 2013/1/7 Konstantin Belousov : >>> I still do remember the buzz about the binary format 0xCAFEBABE, which >>> AFAIR gained image activator support on several OSes, to be garbage >>> collected. >> >> Maybe it would then be a good idea then to add some kind of general >> purpose remapping imgact? Example: >> >> /etc/imgacttab: >> >> cafebabe /usr/local/bin/java >> cffaedfe /usr/local/bin/osx_emulator >> 4243c0de /usr/bin/lli >> >> That way we still give people the freedom to play around with mapping >> their own executable formats, but don't need to maintain a bunch of >> imgacts. > > A generic module that could be somewhat customized at runtime to map > offset+signature into the shebang path could be a possibility indeed. > I strongly prefer to have it as module and not enabled by default. > > Asking Nathan for writing the thing is too much, IMHO, esp. in > the response to the 50-lines hack. > I think this is a good idea, since it both prevents a profusion of similar activators and works nicely in jails and similar environments. I probably won't write it quickly, but it should not take more than about 50 lines, so I can't imagine it will be that bad. There are some complications with this kind of design from the things in the XXX comment in imgact_llvm.c about handling argv[0] that I need to think some more about. Why are you opposed to having it there by default? I think it's actually quite important that it be there by default. Having it not "standard" would be fine, but it should at least be in GENERIC. There are minimal security risks since it just munges begin_argv and doesn't even load the executable and it's little enough code that there should not be any kernel bloat to speak of. If things like this aren't enabled by default, no one can depend on them being there, no one will use it, and the point is entirely lost. -Nathan From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 17:13:14 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3A8FB4C1; Sun, 13 Jan 2013 17:13:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8B36BFB6; Sun, 13 Jan 2013 17:13:13 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DHD48o068424; Sun, 13 Jan 2013 19:13:04 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DHD48o068424 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DHD4HF068423; Sun, 13 Jan 2013 19:13:04 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 19:13:04 +0200 From: Konstantin Belousov To: Nathan Whitehorn Subject: Re: LLVM Image Activator Message-ID: <20130113171304.GZ2561@kib.kiev.ua> References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> <20130113132057.GQ2561@kib.kiev.ua> <50F2DF11.50202@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GEbvMeJjk36GUqgv" Content-Disposition: inline In-Reply-To: <50F2DF11.50202@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 17:13:14 -0000 --GEbvMeJjk36GUqgv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: > On 01/13/13 05:20, Konstantin Belousov wrote: > > On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: > >> Hi Kostik, > >> > >> 2013/1/7 Konstantin Belousov : > >>> I still do remember the buzz about the binary format 0xCAFEBABE, which > >>> AFAIR gained image activator support on several OSes, to be garbage > >>> collected. > >> > >> Maybe it would then be a good idea then to add some kind of general > >> purpose remapping imgact? Example: > >> > >> /etc/imgacttab: > >> > >> cafebabe /usr/local/bin/java > >> cffaedfe /usr/local/bin/osx_emulator > >> 4243c0de /usr/bin/lli > >> > >> That way we still give people the freedom to play around with mapping > >> their own executable formats, but don't need to maintain a bunch of > >> imgacts. > >=20 > > A generic module that could be somewhat customized at runtime to map > > offset+signature into the shebang path could be a possibility indeed. > > I strongly prefer to have it as module and not enabled by default. > >=20 > > Asking Nathan for writing the thing is too much, IMHO, esp. in > > the response to the 50-lines hack. > >=20 >=20 > I think this is a good idea, since it both prevents a profusion of > similar activators and works nicely in jails and similar environments. I > probably won't write it quickly, but it should not take more than about > 50 lines, so I can't imagine it will be that bad. There are some > complications with this kind of design from the things in the XXX > comment in imgact_llvm.c about handling argv[0] that I need to think > some more about. Great. I do not believe in the 50 lines, but I am happy that you want to work this out. >=20 > Why are you opposed to having it there by default? I think it's actually > quite important that it be there by default. Having it not "standard" > would be fine, but it should at least be in GENERIC. There are minimal > security risks since it just munges begin_argv and doesn't even load the > executable and it's little enough code that there should not be any > kernel bloat to speak of. If things like this aren't enabled by default, > no one can depend on them being there, no one will use it, and the point > is entirely lost. All image activators demonstrated a constant stream of security holes. Even our ELF activator, and I was guilty there too. I definitely do not fight over the inclusion of the proposed activator into GENERIC, but do insist on the config option + module. --GEbvMeJjk36GUqgv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8usfAAoJEJDCuSvBvK1B7ZEP/3kcYy5fF7Ld9YBfRAwqMaBu IpEiPPpKtK2KarELSvRXTIYIoy92kJE5ax7Ad+cummm3sNN2yolBQAKJ+fAkHnBv +hWLpKxOmzdnjw4fO0GLU71vNEImTtj8YSymErZxNl10HTrwl8usBkl0SlequI11 m5fbbNNmsBK+6TS9OP/6CN9Pq1exqUPsu4HUDUunFJ+ucOAcCh4tNddcTRZwItc9 wMYErq+XWhd7t29g0PyAH3Tw9h9MgvKPGNrRUzji/Ytno5sv9Xg0ZBQ/MP3PLt4i kulsU3Nvko32YGl/Kme7Kl3jU2sGEq0p9S1IIPqbmyVxd49/3w7pW9SSFffoijPX S+R5MCEKQo3cGvyAEawa0hxaCY84KZd5njGfNN6FQtThwjZMY8bfFNWqAKpbNRmI bXt+sNipZu8vJnEa2NNjr8CtpShvoczkxNMGHVU+ZWiG81QekqdaH8yRSAsgRGB9 T4s0cQTXoI4GmX8Zl6u9p1yFVQ5bS3zs4oCbRUme5zsqHX1L7ufNUl6F8TayELvX 46YOhiuWZIShvI/oRU1MZcgVq6VZy/eElf8WbiiNDKZVyI9VsfJRm6x+b2KM9OlC KdGV4aUDaesjAkk5KQCS118vxZDvtSY7lyksSlN1UqFqq4G+tQi2Ch4I+FvqLV5d kEsyiEh666KnTaWUAZI4 =Xf2/ -----END PGP SIGNATURE----- --GEbvMeJjk36GUqgv-- From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 17:46:32 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E4BEE987; Sun, 13 Jan 2013 17:46:32 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 71FDDFA; Sun, 13 Jan 2013 17:46:31 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id r0DHkUEu058107; Sun, 13 Jan 2013 10:46:30 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r0DHkROi000960; Sun, 13 Jan 2013 10:46:28 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: Fast sigblock (AKA rtld speedup) From: Ian Lepore To: Konstantin Belousov In-Reply-To: <20130113150633.GV2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130112230435.GJ1410@funkthat.com> <1358088614.32417.23.camel@revolution.hippie.lan> <20130113150633.GV2561@kib.kiev.ua> Content-Type: text/plain; charset="us-ascii" Date: Sun, 13 Jan 2013 10:46:27 -0700 Message-ID: <1358099187.32417.31.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, John-Mark Gurney , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 17:46:33 -0000 On Sun, 2013-01-13 at 17:06 +0200, Konstantin Belousov wrote: > On Sun, Jan 13, 2013 at 07:50:14AM -0700, Ian Lepore wrote: > > On Sat, 2013-01-12 at 15:04 -0800, John-Mark Gurney wrote: > > > David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > > > > and can not be freed until process is exited, the page is doubly > > > > mapped into in kernel and userland, accessing the shared data > > > > in kernel has zero overhead though. > > > > > > Don't forget that there are arches out there w/ VIVT caches which will > > > probably eliminate most of the performance benifits if we have the same > > > page mapped writable in two different virtual addresses.. > > > > > > > Even worse than eliminate the benefits, since multiple mappings with one > > writable disables caching on the whole page, there can be a big penalty > > depending on what other data is nearby that suddenly becomes > > uncacheable. I was initially very interested in the work to read system > > clocks without a syscall until I realized it was going to suffer from > > the same problem. > > Since only kernel writes to the shared page, it should be not a problem. > At least there is a specific point where you can insert the neccessary > cache flush commands. With VIVT cache, all mappings of a page must be uncacheable no matter which one is writable. It applies even if all the mappings belong to the kernel (this is why the wired writable pages in the buffer cache kill executable performance with VIVT caches). There's no way to keep a VIVT cache coherent when there are multiple mappings, because of the virtual indexing: you write to a page, then do a flush based on that page's virtual address, and there's no way find cache lines which alias the same physical memory using different virtual addresses to flush them too. > Also, I suspect that even for arms, the writes are executed from the > same core, which could simplify things even more. The arm chips that are affected by this don't have multiple cores (at least, I don't think there are any VIVT cache multi-core arms, it's probably not even possible). -- Ian From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 17:50:43 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 83F6DAB8; Sun, 13 Jan 2013 17:50:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D3151131; Sun, 13 Jan 2013 17:50:42 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0DHoTNB072616; Sun, 13 Jan 2013 19:50:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0DHoTNB072616 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0DHoTb5072615; Sun, 13 Jan 2013 19:50:29 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jan 2013 19:50:29 +0200 From: Konstantin Belousov To: Ian Lepore Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130113175029.GA2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130112230435.GJ1410@funkthat.com> <1358088614.32417.23.camel@revolution.hippie.lan> <20130113150633.GV2561@kib.kiev.ua> <1358099187.32417.31.camel@revolution.hippie.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Qcw+NG9yjQzXdZsc" Content-Disposition: inline In-Reply-To: <1358099187.32417.31.camel@revolution.hippie.lan> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, John-Mark Gurney , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 17:50:43 -0000 --Qcw+NG9yjQzXdZsc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 10:46:27AM -0700, Ian Lepore wrote: > On Sun, 2013-01-13 at 17:06 +0200, Konstantin Belousov wrote: > > On Sun, Jan 13, 2013 at 07:50:14AM -0700, Ian Lepore wrote: > > > On Sat, 2013-01-12 at 15:04 -0800, John-Mark Gurney wrote: > > > > David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > > > > > and can not be freed until process is exited, the page is doubly > > > > > mapped into in kernel and userland, accessing the shared data > > > > > in kernel has zero overhead though. > > > >=20 > > > > Don't forget that there are arches out there w/ VIVT caches which w= ill > > > > probably eliminate most of the performance benifits if we have the = same > > > > page mapped writable in two different virtual addresses.. > > > >=20 > > >=20 > > > Even worse than eliminate the benefits, since multiple mappings with = one > > > writable disables caching on the whole page, there can be a big penal= ty > > > depending on what other data is nearby that suddenly becomes > > > uncacheable. I was initially very interested in the work to read sys= tem > > > clocks without a syscall until I realized it was going to suffer from > > > the same problem. > >=20 > > Since only kernel writes to the shared page, it should be not a problem. > > At least there is a specific point where you can insert the neccessary > > cache flush commands. >=20 > With VIVT cache, all mappings of a page must be uncacheable no matter > which one is writable. It applies even if all the mappings belong to > the kernel (this is why the wired writable pages in the buffer cache > kill executable performance with VIVT caches). >=20 > There's no way to keep a VIVT cache coherent when there are multiple > mappings, because of the virtual indexing: you write to a page, then do > a flush based on that page's virtual address, and there's no way find > cache lines which alias the same physical memory using different virtual > addresses to flush them too. No, you do know all the mapping addresses in this special case. The shared page is mapped at the fixed user address, and at some kernel address. You obviously need to flush both cache lines. >=20 > > Also, I suspect that even for arms, the writes are executed from the > > same core, which could simplify things even more. >=20 > The arm chips that are affected by this don't have multiple cores (at > least, I don't think there are any VIVT cache multi-core arms, it's > probably not even possible). >=20 > -- Ian >=20 --Qcw+NG9yjQzXdZsc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8vPkAAoJEJDCuSvBvK1BPwcP/2J8EpvsPnYRVj9X//KIpxb2 J7a48lmejktht9W7esIMWnBhrYl4bQCWTZuwnffqEwLlTCt5VsHwc1NGfDoigeV2 LQCiFuG/rSkTNpBDBJdNDDxTOmK+wLU6eLck9gj0INtgM7bVQUuP05gfU9utSjvG hCmLIN4nzkiSR9mSZWeJ7n2qq1Qy6k18dT2q68TAiXAscn3uuHxIJXmwLoahLrbb xlxorirXWn62lPFD9VPf0gRW98JzphKY/0oXseI/PhZjKD8ZpoPbnr9kmu1FrBZX Jtezg9eGHd3kH6KGdh1CZ9dxaR54MLcJDQCgN6bxWoJw6BO4hkclq/T6rWpvxOKh 8bXkS8OLsLCH3YYML01PEJtcf6bvkQ3CxuBt+apmkSpVDtRrel0SrT3S9xxPDUQc HD3agGo5FUthJnQzMhqpUHi4S3itA+Gk+3b26Wfza0RIJFp0lXPZ04dXq7RjfiPx B/EEWZb0/GU60OOUH+FoWp6q1jnhrzaOdJcs8pN8TmqpVnXL3z9ErGeOO4rZYspw 2YLYOfB1fpH4mcejqPyEbeSdp1XjUhq3h2CRHVQGjWk0dynosvCnekW/IXsxb4ev OwxUL8NKR3LBj2pnnqbAShB1t+yw3kknl1wuP4ytIaccpAQqizx/sHbw7eDql/ev LP3E/y31ymppF3lj16gz =GCvl -----END PGP SIGNATURE----- --Qcw+NG9yjQzXdZsc-- From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 17:53:22 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 30DECBAE; Sun, 13 Jan 2013 17:53:22 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id F3056158; Sun, 13 Jan 2013 17:53:21 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id r0DHrL7C058501; Sun, 13 Jan 2013 10:53:21 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r0DHrIGw000975; Sun, 13 Jan 2013 10:53:18 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: Fast sigblock (AKA rtld speedup) From: Ian Lepore To: Konstantin Belousov In-Reply-To: <20130113175029.GA2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130112230435.GJ1410@funkthat.com> <1358088614.32417.23.camel@revolution.hippie.lan> <20130113150633.GV2561@kib.kiev.ua> <1358099187.32417.31.camel@revolution.hippie.lan> <20130113175029.GA2561@kib.kiev.ua> Content-Type: text/plain; charset="us-ascii" Date: Sun, 13 Jan 2013 10:53:18 -0700 Message-ID: <1358099598.32417.33.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, John-Mark Gurney , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 17:53:22 -0000 On Sun, 2013-01-13 at 19:50 +0200, Konstantin Belousov wrote: > On Sun, Jan 13, 2013 at 10:46:27AM -0700, Ian Lepore wrote: > > On Sun, 2013-01-13 at 17:06 +0200, Konstantin Belousov wrote: > > > On Sun, Jan 13, 2013 at 07:50:14AM -0700, Ian Lepore wrote: > > > > On Sat, 2013-01-12 at 15:04 -0800, John-Mark Gurney wrote: > > > > > David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > > > > > > and can not be freed until process is exited, the page is doubly > > > > > > mapped into in kernel and userland, accessing the shared data > > > > > > in kernel has zero overhead though. > > > > > > > > > > Don't forget that there are arches out there w/ VIVT caches which will > > > > > probably eliminate most of the performance benifits if we have the same > > > > > page mapped writable in two different virtual addresses.. > > > > > > > > > > > > > Even worse than eliminate the benefits, since multiple mappings with one > > > > writable disables caching on the whole page, there can be a big penalty > > > > depending on what other data is nearby that suddenly becomes > > > > uncacheable. I was initially very interested in the work to read system > > > > clocks without a syscall until I realized it was going to suffer from > > > > the same problem. > > > > > > Since only kernel writes to the shared page, it should be not a problem. > > > At least there is a specific point where you can insert the neccessary > > > cache flush commands. > > > > With VIVT cache, all mappings of a page must be uncacheable no matter > > which one is writable. It applies even if all the mappings belong to > > the kernel (this is why the wired writable pages in the buffer cache > > kill executable performance with VIVT caches). > > > > There's no way to keep a VIVT cache coherent when there are multiple > > mappings, because of the virtual indexing: you write to a page, then do > > a flush based on that page's virtual address, and there's no way find > > cache lines which alias the same physical memory using different virtual > > addresses to flush them too. > No, you do know all the mapping addresses in this special case. > > The shared page is mapped at the fixed user address, and at some > kernel address. You obviously need to flush both cache lines. > > > > > > Also, I suspect that even for arms, the writes are executed from the > > > same core, which could simplify things even more. > > > > The arm chips that are affected by this don't have multiple cores (at > > least, I don't think there are any VIVT cache multi-core arms, it's > > probably not even possible). > > > > -- Ian > > But as soon as you establish the second mapping, the code in pmap.c automatically remaps all existing mappings as uncacheable. It would be some pretty big work to special-case that for a few mappings where the kernel "knows what it's doing." -- ian From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 18:14:22 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E7C94631; Sun, 13 Jan 2013 18:14:22 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from argol.doit.wisc.edu (argol.doit.wisc.edu [144.92.197.212]) by mx1.freebsd.org (Postfix) with ESMTP id C35D2274; Sun, 13 Jan 2013 18:14:22 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from wanderer.tachypleus.net (c-24-63-204-107.hsd1.ct.comcast.net [24.63.204.107]) by smtpauth3.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) with ESMTPSA id <0MGK006GVTBV7J00@smtpauth3.wiscmail.wisc.edu>; Sun, 13 Jan 2013 12:14:22 -0600 (CST) X-Spam-PmxInfo: Server=avs-3, Version=5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2013.1.13.180328, SenderIP=24.63.204.107 X-Spam-Report: AuthenticatedSender=yes, SenderIP=24.63.204.107 X-Wisc-Sender: whitehorn@wisc.edu Message-id: <50F2F97B.5030306@freebsd.org> Date: Sun, 13 Jan 2013 10:14:19 -0800 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130112 Thunderbird/17.0.2 To: Konstantin Belousov Subject: Re: LLVM Image Activator References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> <20130113132057.GQ2561@kib.kiev.ua> <50F2DF11.50202@freebsd.org> <20130113171304.GZ2561@kib.kiev.ua> In-reply-to: <20130113171304.GZ2561@kib.kiev.ua> Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 18:14:23 -0000 On 01/13/13 09:13, Konstantin Belousov wrote: > On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: >> On 01/13/13 05:20, Konstantin Belousov wrote: >>> On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: >>>> Hi Kostik, >>>> >>>> 2013/1/7 Konstantin Belousov : >>>>> I still do remember the buzz about the binary format 0xCAFEBABE, which >>>>> AFAIR gained image activator support on several OSes, to be garbage >>>>> collected. >>>> >>>> Maybe it would then be a good idea then to add some kind of general >>>> purpose remapping imgact? Example: >>>> >>>> /etc/imgacttab: >>>> >>>> cafebabe /usr/local/bin/java >>>> cffaedfe /usr/local/bin/osx_emulator >>>> 4243c0de /usr/bin/lli >>>> >>>> That way we still give people the freedom to play around with mapping >>>> their own executable formats, but don't need to maintain a bunch of >>>> imgacts. >>> >>> A generic module that could be somewhat customized at runtime to map >>> offset+signature into the shebang path could be a possibility indeed. >>> I strongly prefer to have it as module and not enabled by default. >>> >>> Asking Nathan for writing the thing is too much, IMHO, esp. in >>> the response to the 50-lines hack. >>> >> >> I think this is a good idea, since it both prevents a profusion of >> similar activators and works nicely in jails and similar environments. I >> probably won't write it quickly, but it should not take more than about >> 50 lines, so I can't imagine it will be that bad. There are some >> complications with this kind of design from the things in the XXX >> comment in imgact_llvm.c about handling argv[0] that I need to think >> some more about. > Great. I do not believe in the 50 lines, but I am happy that you want > to work this out. > >> >> Why are you opposed to having it there by default? I think it's actually >> quite important that it be there by default. Having it not "standard" >> would be fine, but it should at least be in GENERIC. There are minimal >> security risks since it just munges begin_argv and doesn't even load the >> executable and it's little enough code that there should not be any >> kernel bloat to speak of. If things like this aren't enabled by default, >> no one can depend on them being there, no one will use it, and the point >> is entirely lost. > All image activators demonstrated a constant stream of security holes. > Even our ELF activator, and I was guilty there too. > > I definitely do not fight over the inclusion of the proposed activator > into GENERIC, but do insist on the config option + module. > OK, that sounds like a plan then. I'll try to code up something configurable in the next couple weeks, unless someone else beats me to it. -Nathan From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 13 20:24:41 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D70ED9E4; Sun, 13 Jan 2013 20:24:41 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 93BA99C9; Sun, 13 Jan 2013 20:24:41 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0DKOZpH099223 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 13 Jan 2013 12:24:35 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0DKOZTl099222; Sun, 13 Jan 2013 12:24:35 -0800 (PST) (envelope-from jmg) Date: Sun, 13 Jan 2013 12:24:35 -0800 From: John-Mark Gurney To: Nathan Whitehorn Subject: Re: LLVM Image Activator Message-ID: <20130113202435.GN1410@funkthat.com> Mail-Followup-To: Nathan Whitehorn , Konstantin Belousov , Ed Schouten , freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> <20130113132057.GQ2561@kib.kiev.ua> <50F2DF11.50202@freebsd.org> <20130113171304.GZ2561@kib.kiev.ua> <50F2F97B.5030306@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50F2F97B.5030306@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sun, 13 Jan 2013 12:24:35 -0800 (PST) Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jan 2013 20:24:41 -0000 Nathan Whitehorn wrote this message on Sun, Jan 13, 2013 at 10:14 -0800: > On 01/13/13 09:13, Konstantin Belousov wrote: > > On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: > >> On 01/13/13 05:20, Konstantin Belousov wrote: > >>> On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: > >>>> Hi Kostik, > >>>> > >>>> 2013/1/7 Konstantin Belousov : > >>>>> I still do remember the buzz about the binary format 0xCAFEBABE, which > >>>>> AFAIR gained image activator support on several OSes, to be garbage > >>>>> collected. > >>>> > >>>> Maybe it would then be a good idea then to add some kind of general > >>>> purpose remapping imgact? Example: > >>>> > >>>> /etc/imgacttab: > >>>> > >>>> cafebabe /usr/local/bin/java > >>>> cffaedfe /usr/local/bin/osx_emulator > >>>> 4243c0de /usr/bin/lli > >>>> > >>>> That way we still give people the freedom to play around with mapping > >>>> their own executable formats, but don't need to maintain a bunch of > >>>> imgacts. > >>> > >>> A generic module that could be somewhat customized at runtime to map > >>> offset+signature into the shebang path could be a possibility indeed. > >>> I strongly prefer to have it as module and not enabled by default. > >>> > >>> Asking Nathan for writing the thing is too much, IMHO, esp. in > >>> the response to the 50-lines hack. > >>> > >> > >> I think this is a good idea, since it both prevents a profusion of > >> similar activators and works nicely in jails and similar environments. I > >> probably won't write it quickly, but it should not take more than about > >> 50 lines, so I can't imagine it will be that bad. There are some > >> complications with this kind of design from the things in the XXX > >> comment in imgact_llvm.c about handling argv[0] that I need to think > >> some more about. > > Great. I do not believe in the 50 lines, but I am happy that you want > > to work this out. > > > >> > >> Why are you opposed to having it there by default? I think it's actually > >> quite important that it be there by default. Having it not "standard" > >> would be fine, but it should at least be in GENERIC. There are minimal > >> security risks since it just munges begin_argv and doesn't even load the > >> executable and it's little enough code that there should not be any > >> kernel bloat to speak of. If things like this aren't enabled by default, > >> no one can depend on them being there, no one will use it, and the point > >> is entirely lost. > > All image activators demonstrated a constant stream of security holes. > > Even our ELF activator, and I was guilty there too. > > > > I definitely do not fight over the inclusion of the proposed activator > > into GENERIC, but do insist on the config option + module. > > > > OK, that sounds like a plan then. I'll try to code up something > configurable in the next couple weeks, unless someone else beats me to it. I'll point out that file already has the magic (pun intended) that we are looking for, though I do realize that the code might be a bit much to import.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 16:22:03 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DA8FCB25; Mon, 14 Jan 2013 16:22:03 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id B95D72BB; Mon, 14 Jan 2013 16:22:03 +0000 (UTC) Received: from pakbsde14.localnet (unknown [38.105.238.108]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 177FCB95B; Mon, 14 Jan 2013 11:22:03 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org, toolchain@freebsd.org Subject: Re: Fast sigblock (AKA rtld speedup) Date: Mon, 14 Jan 2013 11:06:07 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p22; KDE/4.5.5; amd64; ; ) References: <20130107182235.GA65279@kib.kiev.ua> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> In-Reply-To: <20130112162547.GA54954@stack.nl> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301141106.07976.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 14 Jan 2013 11:22:03 -0500 (EST) Cc: Jilles Tjoelker X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 16:22:03 -0000 On Saturday, January 12, 2013 11:25:47 am Jilles Tjoelker wrote: > With that, I think fast sigblock is too much code and complication for a > niche case. It does seem a bit complicated to me as well. > Most of the extra atomics in multi-threaded applications are conditional > on __isthreaded (or can be made so); therefore, performance loss from > linking in libthr should be negligible in most cases. Sadly, this is not true. libstdc++ turns on locking if you merely link against libthr, not based on testing __isthreaded. (It does this by testing to see if pthread_once() works during startup, and we have to intentionally sabotage the pthread_once() in libc to fail for this to work which annoys me for an entirely different set of reasons.) At work we go to great lengths to avoid linking in libthr for exactly this reason (e.g. we have a custom port of boost that builds a separate set of boost libraries that are explicitly not linked against libthr), and we also care about exception performance (one of my co-workers submitted the PR about exception performance). -- John Baldwin From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 16:50:12 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 46527811; Mon, 14 Jan 2013 16:50:12 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 331EA648; Mon, 14 Jan 2013 16:50:12 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [64.25.27.130]) by elvis.mu.org (Postfix) with ESMTPSA id 6692E1A3CDA; Mon, 14 Jan 2013 08:50:06 -0800 (PST) Message-ID: <50F4373B.5080203@mu.org> Date: Mon, 14 Jan 2013 11:50:03 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> <201301141106.07976.jhb@freebsd.org> In-Reply-To: <201301141106.07976.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 16:50:12 -0000 On 1/14/13 11:06 AM, John Baldwin wrote: > On Saturday, January 12, 2013 11:25:47 am Jilles Tjoelker wrote: >> With that, I think fast sigblock is too much code and complication for a >> niche case. > It does seem a bit complicated to me as well. > >> Most of the extra atomics in multi-threaded applications are conditional >> on __isthreaded (or can be made so); therefore, performance loss from >> linking in libthr should be negligible in most cases. > Sadly, this is not true. libstdc++ turns on locking if you merely link > against libthr, not based on testing __isthreaded. (It does this by testing > to see if pthread_once() works during startup, and we have to intentionally > sabotage the pthread_once() in libc to fail for this to work which annoys me > for an entirely different set of reasons.) > > At work we go to great lengths to avoid linking in libthr for exactly this > reason (e.g. we have a custom port of boost that builds a separate set of > boost libraries that are explicitly not linked against libthr), and we also > care about exception performance (one of my co-workers submitted the PR about > exception performance). > I get frustrated when people ask me "but why are you doing that?", but I have to know... why do we/you need fast exception handling? Are you throwing a high rate of exceptions? Or is it just that your application is that sensitive to exceptions being thrown that a single slowish one has an impact? -Alfred From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 17:43:52 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 80D80404; Mon, 14 Jan 2013 17:43:52 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 588C78D7; Mon, 14 Jan 2013 17:43:52 +0000 (UTC) Received: from pakbsde14.localnet (unknown [38.105.238.108]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id AE32EB948; Mon, 14 Jan 2013 12:43:51 -0500 (EST) From: John Baldwin To: Alfred Perlstein Subject: Re: Fast sigblock (AKA rtld speedup) Date: Mon, 14 Jan 2013 12:43:40 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p22; KDE/4.5.5; amd64; ; ) References: <20130107182235.GA65279@kib.kiev.ua> <201301141106.07976.jhb@freebsd.org> <50F4373B.5080203@mu.org> In-Reply-To: <50F4373B.5080203@mu.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301141243.40130.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 14 Jan 2013 12:43:51 -0500 (EST) Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 17:43:52 -0000 On Monday, January 14, 2013 11:50:03 am Alfred Perlstein wrote: > On 1/14/13 11:06 AM, John Baldwin wrote: > > On Saturday, January 12, 2013 11:25:47 am Jilles Tjoelker wrote: > >> With that, I think fast sigblock is too much code and complication for a > >> niche case. > > It does seem a bit complicated to me as well. > > > >> Most of the extra atomics in multi-threaded applications are conditional > >> on __isthreaded (or can be made so); therefore, performance loss from > >> linking in libthr should be negligible in most cases. > > Sadly, this is not true. libstdc++ turns on locking if you merely link > > against libthr, not based on testing __isthreaded. (It does this by testing > > to see if pthread_once() works during startup, and we have to intentionally > > sabotage the pthread_once() in libc to fail for this to work which annoys me > > for an entirely different set of reasons.) > > > > At work we go to great lengths to avoid linking in libthr for exactly this > > reason (e.g. we have a custom port of boost that builds a separate set of > > boost libraries that are explicitly not linked against libthr), and we also > > care about exception performance (one of my co-workers submitted the PR about > > exception performance). > > > I get frustrated when people ask me "but why are you doing that?", but I > have to know... why do we/you need fast exception handling? > > Are you throwing a high rate of exceptions? Or is it just that your > application is that sensitive to exceptions being thrown that a single > slowish one has an impact? More that it is sensitive. Not all exceptions result in an immediate call to exit(). -- John Baldwin From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 17:47:19 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F3D556A6; Mon, 14 Jan 2013 17:47:18 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (unknown [IPv6:2001:610:1108:5012::107]) by mx1.freebsd.org (Postfix) with ESMTP id BB73090C; Mon, 14 Jan 2013 17:47:18 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 5F321120400; Mon, 14 Jan 2013 18:47:03 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 4760E2848C; Mon, 14 Jan 2013 18:47:03 +0100 (CET) Date: Mon, 14 Jan 2013 18:47:03 +0100 From: Jilles Tjoelker To: John Baldwin Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130114174703.GB88220@stack.nl> References: <20130107182235.GA65279@kib.kiev.ua> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> <201301141106.07976.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201301141106.07976.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 17:47:19 -0000 On Mon, Jan 14, 2013 at 11:06:07AM -0500, John Baldwin wrote: > On Saturday, January 12, 2013 11:25:47 am Jilles Tjoelker wrote: > > With that, I think fast sigblock is too much code and complication > > for a niche case. > It does seem a bit complicated to me as well. > > Most of the extra atomics in multi-threaded applications are > > conditional on __isthreaded (or can be made so); therefore, > > performance loss from linking in libthr should be negligible in most > > cases. > Sadly, this is not true. libstdc++ turns on locking if you merely > link against libthr, not based on testing __isthreaded. (It does this > by testing to see if pthread_once() works during startup, and we have > to intentionally sabotage the pthread_once() in libc to fail for this > to work which annoys me for an entirely different set of reasons.) > At work we go to great lengths to avoid linking in libthr for exactly > this reason (e.g. we have a custom port of boost that builds a > separate set of boost libraries that are explicitly not linked against > libthr), and we also care about exception performance (one of my > co-workers submitted the PR about exception performance). The code which does that check is actually under contrib/gcc. Problem is, they designed __gthread_active_p() to distinguish threaded and unthreaded programming environments -- it must be known in advance and cannot be changed later. The code for the unthreaded environment then takes advantage of this by not even allocating memory for mutexes in some cases. So there is no way around __gthread_active_p() returning true if libthr is linked in. The __isthreaded check will have to be in libthr or in contrib/gcc/gthr-posix.c and it will not do much more than avoiding atomics. This __gthread_active_p() thing is another barrier to bringing in a threaded plugin in an unthreaded application. Ports people spend a fair amount of time adding -pthread flags to things (such as perl) to work around this. -- Jilles Tjoelker From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 18:24:14 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0730C6BC; Mon, 14 Jan 2013 18:24:14 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from theravensnest.org (theraven.freebsd.your.org [216.14.102.27]) by mx1.freebsd.org (Postfix) with ESMTP id D0645B0C; Mon, 14 Jan 2013 18:24:13 +0000 (UTC) Received: from [192.168.0.2] (cpc10-cmbg15-2-0-cust123.5-4.cable.virginmedia.com [86.30.246.124]) (authenticated bits=0) by theravensnest.org (8.14.5/8.14.5) with ESMTP id r0EIOATH014273 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Mon, 14 Jan 2013 18:24:12 GMT (envelope-from theraven@FreeBSD.org) Subject: Re: Fast sigblock (AKA rtld speedup) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/signed; boundary="Apple-Mail=_70750EDA-D886-486C-9220-8B45D4AB7DD4"; protocol="application/pgp-signature"; micalg=pgp-sha1 From: David Chisnall In-Reply-To: <20130114174703.GB88220@stack.nl> Date: Mon, 14 Jan 2013 18:24:04 +0000 Message-Id: References: <20130107182235.GA65279@kib.kiev.ua> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> <201301141106.07976.jhb@freebsd.org> <20130114174703.GB88220@stack.nl> To: Jilles Tjoelker X-Mailer: Apple Mail (2.1278) Cc: toolchain@FreeBSD.org, John Baldwin , freebsd-arch@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 18:24:14 -0000 --Apple-Mail=_70750EDA-D886-486C-9220-8B45D4AB7DD4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 14 Jan 2013, at 17:47, Jilles Tjoelker wrote: > The code which does that check is actually under contrib/gcc. Problem > is, they designed __gthread_active_p() to distinguish threaded and > unthreaded programming environments -- it must be known in advance and > cannot be changed later. The code for the unthreaded environment then > takes advantage of this by not even allocating memory for mutexes in > some cases. It's worth taking a step back and asking why this code exists at all, = and the main reason is that acquiring a mutex used to be really = expensive. It still is on some fruit-flavoured operating systems, but = elsewhere it's a single atomic operation in the uncontended case, and in = that case the cache line will already be exclusively owned by the = calling core in single-threaded code. =20 I would much rather that we followed the example of Solaris and made the = multithreaded case fast and the default than keep piling on hacks that = allow code to shave off a few clock cycles in the single-threaded case. = In particular, the popularity of multicore systems means that it is = increasingly rare for code to be both single threaded and performance = critical, so this seems like misplaced optimisation. I strongly suspect that making it possible to inline the uncontended = lock case for a pthread mutex and eliminating all of the branches on = __isthreaded would give us a net speedup in both single and = multithreaded cases. > This __gthread_active_p() thing is another barrier to bringing in a > threaded plugin in an unthreaded application. Ports people spend a = fair > amount of time adding -pthread flags to things (such as perl) to work > around this. This and the similar checks in libc cause a lot of pain, and it seems = that the correct fix is ensuring that the performance penalty for = linking libthr is so small that there is no point in avoiding it. David --Apple-Mail=_70750EDA-D886-486C-9220-8B45D4AB7DD4 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) iQIcBAEBAgAGBQJQ9E1FAAoJEKx65DEEsqIdUgcP/RQ4sF0gYsuCGiWUlmiioKnE 3ZQF76ieurN7Hbmq0WNs3eAHljFzQMRPrsgqQVcbyTNPuIuSX4zIdTGLLFwBijf0 X2R0nO6e7sHTYKtCcHmXFoH7DCoQSEG88F1q7zRA1RlvOF0hXDXHEYrSCpWeBMnC 5SwcYMgsZ5eXX9a5tvsUeq2/GyDcPYEkVhq3ueZRmVxIXoaL5Eq3qZ4hReJCLo/1 AnB+/c0dAMJQE6td8gdn7+8EcbHeAblGvpRJFYaNT56WiAVbu+ZOB1l2wNRzMM3e mYsg72pfUxcqb6WWwgk4pXqPQyIMT9pHCwden2rrEpzk7qHFQUV3odVyo2SXtA44 xMWBs2d2a8fmMRCW6wrtrpb1jlPo9W4KmQWpF+4Kaq2P8DuN0ljyTRSC5PQqM4ms saFYl6OOtRFPzD/6RUddklQIi2poBhVp6hAfA2qxq0otMN1ZmkpTsRtsNZltXbpp 9fyeHpc2IsBx9uM7ND2b5FQmdXKq1Zs0sF2HC3uhH2Q7F2r39TuM/0m5eayyJosZ bWExLzQmq5gpR6guEEV4pdgye33eCL1TvVgRGOPxmpenydqEyFiflcu16bh5wRU2 DkMJGe6r9OBKqnvNOrlrtE9P16906C9XL9QwUHfnjg440/WAFW2i44zj0U+PE3Bf +GwlZRaF3NgeTX7i7nmH =ADYu -----END PGP SIGNATURE----- --Apple-Mail=_70750EDA-D886-486C-9220-8B45D4AB7DD4-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 18:56:50 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 76CA72E0; Mon, 14 Jan 2013 18:56:50 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id BD1D8DD4; Mon, 14 Jan 2013 18:56:49 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id r0EIulEp016642; Mon, 14 Jan 2013 12:56:47 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id r0EIukBU016641; Mon, 14 Jan 2013 12:56:46 -0600 (CST) (envelope-from brooks) Date: Mon, 14 Jan 2013 12:56:46 -0600 From: Brooks Davis To: Nathan Whitehorn , Konstantin Belousov , Ed Schouten , freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org Subject: Re: LLVM Image Activator Message-ID: <20130114185646.GB15933@lor.one-eyed-alien.net> References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> <20130107172433.GX82219@kib.kiev.ua> <20130113132057.GQ2561@kib.kiev.ua> <50F2DF11.50202@freebsd.org> <20130113171304.GZ2561@kib.kiev.ua> <50F2F97B.5030306@freebsd.org> <20130113202435.GN1410@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tsOsTdHNUZQcU9Ye" Content-Disposition: inline In-Reply-To: <20130113202435.GN1410@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 18:56:50 -0000 --tsOsTdHNUZQcU9Ye Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 13, 2013 at 12:24:35PM -0800, John-Mark Gurney wrote: > Nathan Whitehorn wrote this message on Sun, Jan 13, 2013 at 10:14 -0800: > > On 01/13/13 09:13, Konstantin Belousov wrote: > > > On Sun, Jan 13, 2013 at 08:21:37AM -0800, Nathan Whitehorn wrote: > > >> On 01/13/13 05:20, Konstantin Belousov wrote: > > >>> On Sun, Jan 13, 2013 at 12:41:09PM +0100, Ed Schouten wrote: > > >>>> Hi Kostik, > > >>>> > > >>>> 2013/1/7 Konstantin Belousov : > > >>>>> I still do remember the buzz about the binary format 0xCAFEBABE, = which > > >>>>> AFAIR gained image activator support on several OSes, to be garba= ge > > >>>>> collected. > > >>>> > > >>>> Maybe it would then be a good idea then to add some kind of general > > >>>> purpose remapping imgact? Example: > > >>>> > > >>>> /etc/imgacttab: > > >>>> > > >>>> cafebabe /usr/local/bin/java > > >>>> cffaedfe /usr/local/bin/osx_emulator > > >>>> 4243c0de /usr/bin/lli > > >>>> > > >>>> That way we still give people the freedom to play around with mapp= ing > > >>>> their own executable formats, but don't need to maintain a bunch of > > >>>> imgacts. > > >>> > > >>> A generic module that could be somewhat customized at runtime to map > > >>> offset+signature into the shebang path could be a possibility indee= d. > > >>> I strongly prefer to have it as module and not enabled by default. > > >>> > > >>> Asking Nathan for writing the thing is too much, IMHO, esp. in > > >>> the response to the 50-lines hack. > > >>> > > >> > > >> I think this is a good idea, since it both prevents a profusion of > > >> similar activators and works nicely in jails and similar environment= s. I > > >> probably won't write it quickly, but it should not take more than ab= out > > >> 50 lines, so I can't imagine it will be that bad. There are some > > >> complications with this kind of design from the things in the XXX > > >> comment in imgact_llvm.c about handling argv[0] that I need to think > > >> some more about. > > > Great. I do not believe in the 50 lines, but I am happy that you want > > > to work this out. > > >=20 > > >> > > >> Why are you opposed to having it there by default? I think it's actu= ally > > >> quite important that it be there by default. Having it not "standard" > > >> would be fine, but it should at least be in GENERIC. There are minim= al > > >> security risks since it just munges begin_argv and doesn't even load= the > > >> executable and it's little enough code that there should not be any > > >> kernel bloat to speak of. If things like this aren't enabled by defa= ult, > > >> no one can depend on them being there, no one will use it, and the p= oint > > >> is entirely lost. > > > All image activators demonstrated a constant stream of security holes. > > > Even our ELF activator, and I was guilty there too. > > >=20 > > > I definitely do not fight over the inclusion of the proposed activator > > > into GENERIC, but do insist on the config option + module. > > >=20 > >=20 > > OK, that sounds like a plan then. I'll try to code up something > > configurable in the next couple weeks, unless someone else beats me to = it. >=20 > I'll point out that file already has the magic (pun intended) that we > are looking for, though I do realize that the code might be a bit much > to import.. As someone who recently stuffed libmagic into a very constrained sandbox environment, I can safely assert that you don't want to go there. The code isn't written in a way that would make this easy and I definitely wouldn't want it in the kernel. -- Brooks --tsOsTdHNUZQcU9Ye Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQ9FTtXY6L6fI4GtQRAv4mAKCe6ty9ESLLQmvKYE5JETr9ATOMNgCgg8D+ 9qYBdes15mbVGBbdHm8A+Ds= =Rf1J -----END PGP SIGNATURE----- --tsOsTdHNUZQcU9Ye-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 14 19:37:07 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B4962623; Mon, 14 Jan 2013 19:37:07 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 8FB1534A; Mon, 14 Jan 2013 19:37:07 +0000 (UTC) Received: from pakbsde14.localnet (unknown [38.105.238.108]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A04EBB95B; Mon, 14 Jan 2013 14:37:06 -0500 (EST) From: John Baldwin To: David Chisnall Subject: Re: Fast sigblock (AKA rtld speedup) Date: Mon, 14 Jan 2013 13:58:32 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p22; KDE/4.5.5; amd64; ; ) References: <20130107182235.GA65279@kib.kiev.ua> <20130114174703.GB88220@stack.nl> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201301141358.33216.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 14 Jan 2013 14:37:06 -0500 (EST) Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2013 19:37:07 -0000 On Monday, January 14, 2013 1:24:04 pm David Chisnall wrote: > On 14 Jan 2013, at 17:47, Jilles Tjoelker wrote: > > > The code which does that check is actually under contrib/gcc. Problem > > is, they designed __gthread_active_p() to distinguish threaded and > > unthreaded programming environments -- it must be known in advance and > > cannot be changed later. The code for the unthreaded environment then > > takes advantage of this by not even allocating memory for mutexes in > > some cases. > > It's worth taking a step back and asking why this code exists at all, and the main reason is that acquiring a mutex used to be really expensive. It still is on some fruit-flavoured operating systems, but elsewhere it's a single atomic operation in the uncontended case, and in that case the cache line will already be exclusively owned by the calling core in single-threaded code. > > I would much rather that we followed the example of Solaris and made the multithreaded case fast and the default than keep piling on hacks that allow code to shave off a few clock cycles in the single-threaded case. In particular, the popularity of multicore systems means that it is increasingly rare for code to be both single threaded and performance critical, so this seems like misplaced optimisation. We have single-threaded performance critical applications that run on multicore systems (we just run several copies) and if we link in libthr, then pthread_mutex operations (even on uncontested locks) show up as one of the top consumers of CPU time when we profile our applications. > I strongly suspect that making it possible to inline the uncontended lock case for a pthread mutex and eliminating all of the branches on __isthreaded would give us a net speedup in both single and multithreaded cases. I'm less certain. Note that you can't inline mutex ops until you expose the mutexes themselves to userland (that is, making pthread_mutex_t not be opaque). -- John Baldwin From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 15 09:21:37 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A2039B36; Tue, 15 Jan 2013 09:21:37 +0000 (UTC) (envelope-from theraven@freebsd.org) Received: from theravensnest.org (theraven.freebsd.your.org [216.14.102.27]) by mx1.freebsd.org (Postfix) with ESMTP id 5747D78F; Tue, 15 Jan 2013 09:21:37 +0000 (UTC) Received: from [192.168.0.2] (cpc10-cmbg15-2-0-cust123.5-4.cable.virginmedia.com [86.30.246.124]) (authenticated bits=0) by theravensnest.org (8.14.5/8.14.5) with ESMTP id r0F9LThB017760 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Tue, 15 Jan 2013 09:21:30 GMT (envelope-from theraven@freebsd.org) Subject: Re: Fast sigblock (AKA rtld speedup) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: David Chisnall In-Reply-To: <201301141358.33216.jhb@freebsd.org> Date: Tue, 15 Jan 2013 09:21:25 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20130107182235.GA65279@kib.kiev.ua> <20130114174703.GB88220@stack.nl> <201301141358.33216.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1278) Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2013 09:21:37 -0000 On 14 Jan 2013, at 18:58, John Baldwin wrote: > I'm less certain. Note that you can't inline mutex ops until you = expose > the mutexes themselves to userland (that is, making pthread_mutex_t = not > be opaque). This is one of the things that will be required anyway if we wish to = support process-shared mutexes (they've been in POSIX since 1997, so = it's probably getting on for time we did), as the current = mutex-is-a-pointer implementation depends on the virtual address space = of the creator, and so does not work if the mutex is created in a shared = memory segment. That said, even with the current implementation we wouldn't need to = expose the entire mutex structure, just the word that is used as the = fast path. The inline version would be something like: struct pthread_mutex_header { _Atomic(long) lock_word; // other private fields not exposed in header; }; typedef struct pthread_mutex_header *pthread_mutex_t; // Implementation in libthr / libc, which calls into the kernel. int pthread_mutex_lock_slowly(pthread_mutex_t*); inline int pthread_mutex_lock(pthread_mutex_t *mutex) { int desired =3D 0; if (atomic_compare_exchange_weak_explicit(&(*mutex)->lock_word, = &desired, 1, memory_order_acquire, memory_order_relaxed)) return 0; return pthread_mutex_lock_slowly(mutex); } The slow path is only needed when the mutex can't be acquired trivially = in userspace. On x86, the fast path adds 6 extra instructions, including = a branch that can be statically hinted if we want (assume that we won't = be going down the slow path, because a mispredicted branch doesn't add = much to the cost of the syscall if we are). =20 The corresponding saving is that we get to delete a massive pile of = conditionals that we currently have for __is_threaded. We'd also = completely avoid the function call (which is actually two function = calls, as we do some trampoline things in libc) in the fast-path case = for threaded applications. A similar saving is possible with read-write locks and possibly with = condition variables, although our kernel interface for these is = incredibly poorly documented (for once, Linux actually has better = documentation: futexes are very well documented). Looking in umtx.h, it = sort-of exposes inline functions that look like this, but given the = complete lack of documentation, I have no idea how useable they are. =20 David= From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 15 19:35:50 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7817A200 for ; Tue, 15 Jan 2013 19:35:50 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 58C7E153 for ; Tue, 15 Jan 2013 19:35:50 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0FJZnkn043923 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 15 Jan 2013 11:35:49 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0FJZnnp043922 for toolchain@FreeBSD.org; Tue, 15 Jan 2013 11:35:49 -0800 (PST) (envelope-from jmg) Date: Tue, 15 Jan 2013 11:35:49 -0800 From: John-Mark Gurney To: toolchain@FreeBSD.org Subject: temp file turd left behind in odd case... Message-ID: <20130115193549.GB1410@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 15 Jan 2013 11:35:50 -0800 (PST) X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2013 19:35:50 -0000 If you try to compile the following file: #include __m128i bar(__m128i a, __m128i b) { return _mm_aesenc_si128(a, b); } w/ the command: clang -D__AES__ -c intrintest.c You'll get the error: fatal error: error in backend: Cannot select: intrinsic %llvm.x86.aesni.aesenc and a intrintest.o-XXXXXXXX temp file left behind from the failed compile... Yes, the correct way to enable the AES instructions is with the -maes option, but I didn't know that at the time... Looks like clang doesn't fully clean up in this case... I tried this on MacOSX's clang, but the smmintrin.h header prevents me from compiling w/ the error: In file included from /usr/bin/../lib/clang/2.1/include/wmmintrin.h:31: /usr/bin/../lib/clang/2.1/include/smmintrin.h:28:2: error: #error "SSE4.1 instruction set not enabled" #error "SSE4.1 instruction set not enabled" It'd be nice if -maes and related SSE enabling options were documented in the man page... :) P.S. Not on the list, so CC me if necessary. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 15 20:05:21 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CA87DC35 for ; Tue, 15 Jan 2013 20:05:21 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7D0612EA for ; Tue, 15 Jan 2013 20:05:21 +0000 (UTC) Received: from [192.168.0.6] (spaceball.home.andric.com [192.168.0.6]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 0A1FB5C37; Tue, 15 Jan 2013 21:05:20 +0100 (CET) Message-ID: <50F5B67C.3000209@FreeBSD.org> Date: Tue, 15 Jan 2013 21:05:16 +0100 From: Dimitry Andric Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20121128 Thunderbird/18.0 MIME-Version: 1.0 To: John-Mark Gurney Subject: Re: temp file turd left behind in odd case... References: <20130115193549.GB1410@funkthat.com> In-Reply-To: <20130115193549.GB1410@funkthat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2013 20:05:21 -0000 On 2013-01-15 20:35, John-Mark Gurney wrote: > If you try to compile the following file: > #include > > __m128i > bar(__m128i a, __m128i b) > { > return _mm_aesenc_si128(a, b); > } > > w/ the command: > clang -D__AES__ -c intrintest.c > > You'll get the error: > fatal error: error in backend: Cannot select: intrinsic %llvm.x86.aesni.aesenc > > and a intrintest.o-XXXXXXXX temp file left behind from the failed compile... Hm, weird... On i386, this simply produces errors about SSE2 not being enabled, and a whole bunch of other errors and warnings. On amd64, it does indeed produce the llvm backend error you have shown. I will report this to upstream, but there is a good chance they will say "don't do that then". :-) > Yes, the correct way to enable the AES instructions is with the -maes > option, but I didn't know that at the time... Looks like clang doesn't > fully clean up in this case... > > I tried this on MacOSX's clang, but the smmintrin.h header prevents me > from compiling w/ the error: > In file included from /usr/bin/../lib/clang/2.1/include/wmmintrin.h:31: > /usr/bin/../lib/clang/2.1/include/smmintrin.h:28:2: error: #error "SSE4.1 > instruction set not enabled" > #error "SSE4.1 instruction set not enabled" > > It'd be nice if -maes and related SSE enabling options were documented > in the man page... :) We don't maintain our own manpage, upstream produces either .rst or .pod files, and manpages or other docs are generated from those. Please send any suggestions for documentation updates to one of the appropriate upstream mailing lists for review. From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 15 20:27:58 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 98D3F8BA; Tue, 15 Jan 2013 20:27:58 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 71DA0645; Tue, 15 Jan 2013 20:27:58 +0000 (UTC) Received: from pakbsde14.localnet (unknown [38.105.238.108]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id CA5E0B918; Tue, 15 Jan 2013 15:27:57 -0500 (EST) From: John Baldwin To: David Chisnall Subject: Re: Fast sigblock (AKA rtld speedup) Date: Tue, 15 Jan 2013 14:45:26 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p22; KDE/4.5.5; amd64; ; ) References: <20130107182235.GA65279@kib.kiev.ua> <201301141358.33216.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301151445.26895.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 15 Jan 2013 15:27:57 -0500 (EST) Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2013 20:27:58 -0000 On Tuesday, January 15, 2013 4:21:25 am David Chisnall wrote: > On 14 Jan 2013, at 18:58, John Baldwin wrote: > > > I'm less certain. Note that you can't inline mutex ops until you expose > > the mutexes themselves to userland (that is, making pthread_mutex_t not > > be opaque). > > This is one of the things that will be required anyway if we wish to support process-shared mutexes (they've been in POSIX since 1997, so it's probably getting on for time we did), as the current mutex-is-a-pointer implementation depends on the virtual address space of the creator, and so does not work if the mutex is created in a shared memory segment. Yes, David Xu has a p4 branch with this done already. My point is that I would rather effort be spent on getting that in before attempting your suggestion for our current primitives as the inlining you do now requires that David's changes honor the same ABI in the future. -- John Baldwin From owner-freebsd-toolchain@FreeBSD.ORG Wed Jan 16 00:30:39 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 099CD997; Wed, 16 Jan 2013 00:30:39 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from mail-pa0-f46.google.com (mail-pa0-f46.google.com [209.85.220.46]) by mx1.freebsd.org (Postfix) with ESMTP id A8B8189A; Wed, 16 Jan 2013 00:30:38 +0000 (UTC) Received: by mail-pa0-f46.google.com with SMTP id bh2so433900pad.5 for ; Tue, 15 Jan 2013 16:30:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:reply-to:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=cYvtlBtpMeYM3HqF+sARSGxitLS5o0WxtfxLkJus1TE=; b=uu3F3jpjuMOxPGmbGLSULzU836KHfdkJCq7xYoygksrjvd8c3r90BC01fygqQFqsIx OeMyEkZqhnf78aWKza+QSZptZZOPDDWk0kD6JLa2ecyoL2FhITZw5spKujJJuUq2nLDN zwJ+Pv7fhSfsbHKELvNVCQxrRIuvpxp6GYZFRdJEO9Yr4Ld14CN7TEqhoPIzQ0G3EN8h h4svv6wTEark7zySAJ4CGyaJZx+7MxPuVLiIP+QxPr+Tg5NbnI19tiIDSRtCB66mv1hn ZzosoQtgyEqcF8FGfU49YjomFz9EDOQqUqksxjZ5DHfRopGJrGNBk6/8Va9Z4/3x/Kj7 Ec0A== X-Received: by 10.68.252.4 with SMTP id zo4mr273293689pbc.126.1358296233913; Tue, 15 Jan 2013 16:30:33 -0800 (PST) Received: from xp5k.my.domain ([115.192.139.240]) by mx.google.com with ESMTPS id kl3sm11114067pbc.15.2013.01.15.16.30.29 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Jan 2013 16:30:32 -0800 (PST) Message-ID: <50F5F4A2.6080200@gmail.com> Date: Wed, 16 Jan 2013 08:30:26 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <201301141358.33216.jhb@freebsd.org> <201301151445.26895.jhb@freebsd.org> In-Reply-To: <201301151445.26895.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: toolchain@freebsd.org, Jilles Tjoelker , freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: davidxu@freebsd.org List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 00:30:39 -0000 On 2013/01/16 03:45, John Baldwin wrote: > On Tuesday, January 15, 2013 4:21:25 am David Chisnall wrote: >> On 14 Jan 2013, at 18:58, John Baldwin wrote: >> >>> I'm less certain. Note that you can't inline mutex ops until you expose >>> the mutexes themselves to userland (that is, making pthread_mutex_t not >>> be opaque). >> This is one of the things that will be required anyway if we wish to support > process-shared mutexes (they've been in POSIX since 1997, so it's probably > getting on for time we did), as the current mutex-is-a-pointer implementation > depends on the virtual address space of the creator, and so does not work if > the mutex is created in a shared memory segment. > > Yes, David Xu has a p4 branch with this done already. My point is that I > would rather effort be spent on getting that in before attempting your > suggestion for our current primitives as the inlining you do now requires > that David's changes honor the same ABI in the future. > Yes, I have such a p4 branch. The problem I encountered is binary compatibility, libthr has to have both code for old mutex which is a pointer and new mutex which is a structure, if module A passes pointer-mutex to a recompiled module B which is using structure-mutex, this is broken. And I found NVIDIA GeForce driver is using dlsym() with no version hint, it is trying to get pthread_mutex_lock and other symbols, the default version in the new libthr will return a new entry which uses structure mutex, since the driver is not open-source, so you can not recompile it, the change will break it. Regards, David Xu From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 07:05:24 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 057BD775 for ; Thu, 17 Jan 2013 07:05:24 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id DCCD38CA for ; Thu, 17 Jan 2013 07:05:23 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0H75Hdl071925 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 16 Jan 2013 23:05:17 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0H75HDK071924 for toolchain@FreeBSD.org; Wed, 16 Jan 2013 23:05:17 -0800 (PST) (envelope-from jmg) Date: Wed, 16 Jan 2013 23:05:16 -0800 From: John-Mark Gurney To: toolchain@FreeBSD.org Subject: patch to add aes and pclmulqdq instructions to gcc Message-ID: <20130117070516.GI1410@funkthat.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="IU5/I01NYhRvwH70" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 16 Jan 2013 23:05:17 -0800 (PST) X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 07:05:24 -0000 --IU5/I01NYhRvwH70 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Mike Belopuhov pointed me to the patch in OpenBSD: http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef704629e/diff.txt While OpenBSD's binutils is quite different than FreeBSD's, I was able to use his patch to teach binutils how to assemble and disassemble the aes and pclmulqdq instructions. I have done basic tests, such as verified that it can assemble the aesni module and get the same results, and assemble a sample file for pclmulqdq.. For each of these tests, I have verified that it's output matches (as close as possible, as gcc/clang compile callq's differently) clang on amd64.. I have attached the patch, and it is also availble at: http://people.freebsd.org/~jmg/gcc.aes.patch Comments? I have not passed it through a make universe yet, but will before committing... I am also working on basic intrinsics header files for these instructions too... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --IU5/I01NYhRvwH70 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="gcc.aes.patch" Index: contrib/binutils/gas/config/tc-i386.c =================================================================== --- contrib/binutils/gas/config/tc-i386.c (revision 245361) +++ contrib/binutils/gas/config/tc-i386.c (working copy) @@ -3981,7 +3981,7 @@ SSE4 instructions have 3 bytes. We may use one more higher byte to specify a prefix the instruction requires. Exclude instructions which are in both SSE4 and ABM. */ - if ((i.tm.cpu_flags & (CpuSSSE3 | CpuSSE4)) != 0 + if ((i.tm.cpu_flags & (CpuSSSE3 | CpuSSE4 | CpuAES | CpuPCLMUL)) != 0 && (i.tm.cpu_flags & CpuABM) == 0) { if (i.tm.base_opcode & 0xff000000) @@ -4033,7 +4033,7 @@ } else { - if ((i.tm.cpu_flags & (CpuSSSE3 | CpuSSE4)) != 0 + if ((i.tm.cpu_flags & (CpuSSSE3 | CpuSSE4 | CpuAES | CpuPCLMUL)) != 0 && (i.tm.cpu_flags & CpuABM) == 0) { p = frag_more (3); Index: contrib/binutils/opcodes/i386-dis.c =================================================================== --- contrib/binutils/opcodes/i386-dis.c (revision 245361) +++ contrib/binutils/opcodes/i386-dis.c (working copy) @@ -543,6 +543,13 @@ #define PREGRP97 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 97 } } #define PREGRP98 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 98 } } #define PREGRP99 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 99 } } +#define PREGRP100 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 100 } } +#define PREGRP101 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 101 } } +#define PREGRP102 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 102 } } +#define PREGRP103 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 103 } } +#define PREGRP104 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 104 } } +#define PREGRP105 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 105 } } +#define PREGRP106 NULL, { { NULL, USE_PREFIX_USER_TABLE }, { NULL, 106 } } #define X86_64_0 NULL, { { NULL, X86_64_SPECIAL }, { NULL, 0 } } @@ -1319,7 +1326,7 @@ /* a0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* af */ /* b0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* bf */ /* c0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* cf */ - /* d0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* df */ + /* d0 */ 0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1, /* df */ /* e0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* ef */ /* f0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* ff */ /* ------------------------------- */ @@ -1382,7 +1389,7 @@ /* 10 */ 0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0, /* 1f */ /* 20 */ 1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, /* 2f */ /* 30 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* 3f */ - /* 40 */ 1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, /* 4f */ + /* 40 */ 1,1,1,0,1,0,0,0,0,0,0,0,0,0,0,0, /* 4f */ /* 50 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* 5f */ /* 60 */ 1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0, /* 6f */ /* 70 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* 7f */ @@ -1391,7 +1398,7 @@ /* a0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* af */ /* b0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* bf */ /* c0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* cf */ - /* d0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* df */ + /* d0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1, /* df */ /* e0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* ef */ /* f0 */ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, /* ff */ /* ------------------------------- */ @@ -2605,6 +2612,62 @@ { "invvpid",{ Gm, Mo } }, { "(bad)", { XX } }, }, + + /* PREGRP100 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aesimc", { XM, EXx } }, + { "(bad)", { XX } }, + }, + + /* PREGRP101 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aesenc",{ XM, EXx } }, + { "(bad)", { XX } }, + }, + + /* PREGRP102 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aesenclast", { XM, EXx } }, + { "(bad)", { XX } }, + }, + + /* PREGRP103 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aesdec", { XM, EXx } }, + { "(bad)", { XX } }, + }, + + /* PREGRP104 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aesdeclast", { XM, EXx } }, + { "(bad)", { XX } }, + }, + + /* PREGRP105 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "aeskeygenassist", { XM, EXx, Ib } }, + { "(bad)", { XX } }, + }, + + /* PREGRP106 */ + { + { "(bad)", { XX } }, + { "(bad)", { XX } }, + { "pclmulqdq", { XM, EXx, Ib } }, + { "(bad)", { XX } }, + }, }; static const struct dis386 x86_64_table[][2] = { @@ -2876,11 +2939,11 @@ { "(bad)", { XX } }, { "(bad)", { XX } }, { "(bad)", { XX } }, - { "(bad)", { XX } }, - { "(bad)", { XX } }, - { "(bad)", { XX } }, - { "(bad)", { XX } }, - { "(bad)", { XX } }, + { PREGRP100 }, + { PREGRP101 }, + { PREGRP102 }, + { PREGRP103 }, + { PREGRP104 }, /* e0 */ { "(bad)", { XX } }, { "(bad)", { XX } }, @@ -2997,10 +3060,10 @@ { PREGRP84 }, { PREGRP85 }, { "(bad)", { XX } }, + { PREGRP106 }, { "(bad)", { XX } }, { "(bad)", { XX } }, { "(bad)", { XX } }, - { "(bad)", { XX } }, /* 48 */ { "(bad)", { XX } }, { "(bad)", { XX } }, @@ -3171,7 +3234,7 @@ { "(bad)", { XX } }, { "(bad)", { XX } }, { "(bad)", { XX } }, - { "(bad)", { XX } }, + { PREGRP105 }, /* e0 */ { "(bad)", { XX } }, { "(bad)", { XX } }, Index: contrib/binutils/opcodes/i386-opc.h =================================================================== --- contrib/binutils/opcodes/i386-opc.h (revision 245361) +++ contrib/binutils/opcodes/i386-opc.h (working copy) @@ -72,6 +72,8 @@ #define CpuSSE4_1 0x400000 /* SSE4.1 Instructions required */ #define CpuSSE4_2 0x800000 /* SSE4.2 Instructions required */ #define CpuXSAVE 0x1000000 /* XSAVE Instructions required */ +#define CpuAES 0x2000000 /* AES Instructions required */ +#define CpuPCLMUL 0x4000000 /* Carry-less Multiplication extensions */ /* SSE4.1/4.2 Instructions required */ #define CpuSSE4 (CpuSSE4_1|CpuSSE4_2) @@ -84,7 +86,7 @@ #define CpuUnknownFlags (Cpu186|Cpu286|Cpu386|Cpu486|Cpu586|Cpu686 \ |CpuP4|CpuSledgehammer|CpuMMX|CpuMMX2|CpuSSE|CpuSSE2|CpuSSE3|CpuVMX \ |Cpu3dnow|Cpu3dnowA|CpuK6|CpuPadLock|CpuSVME|CpuSSSE3|CpuSSE4_1 \ - |CpuSSE4_2|CpuABM|CpuSSE4a|CpuXSAVE) + |CpuSSE4_2|CpuABM|CpuSSE4a|CpuXSAVE|CpuAES|CpuPCLMUL) /* the bits in opcode_modifier are used to generate the final opcode from the base_opcode. These bits also are used to detect alternate forms of @@ -126,6 +128,8 @@ #define Rex64 0x10000000 /* instruction require Rex64 prefix. */ #define Ugh 0x20000000 /* deprecated fp insn, gets a warning */ +#define NoSuf (No_bSuf|No_wSuf|No_lSuf|No_sSuf|No_qSuf|No_xSuf) + /* operand_types[i] describes the type of operand i. This is made by OR'ing together all of the possible type masks. (e.g. 'operand_types[i] = Reg|Imm' specifies that operand i can be Index: contrib/binutils/opcodes/i386-tbl.h =================================================================== --- contrib/binutils/opcodes/i386-tbl.h (revision 245361) +++ contrib/binutils/opcodes/i386-tbl.h (working copy) @@ -4319,6 +4319,54 @@ { "xrstor", 1, 0xfae, 0x5, CpuXSAVE, Modrm|No_bSuf|No_wSuf|No_lSuf|No_sSuf|No_xSuf, { BaseIndex|Disp8|Disp16|Disp32|Disp32S } }, + /* Intel AES extensions */ + {"aesdec", 2, 0x660f38de, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { RegXMM|LLongMem, + RegXMM } }, + {"aesdeclast", 2, 0x660f38df, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { RegXMM|LLongMem, + RegXMM } }, + {"aesenc", 2, 0x660f38dc, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { RegXMM|LLongMem, + RegXMM } }, + {"aesenclast", 2, 0x660f38dd, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { RegXMM|LLongMem, + RegXMM } }, + {"aesimc", 2, 0x660f38db, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { RegXMM|LLongMem, + RegXMM } }, + {"aeskeygenassist", 3, 0x660f3adf, None, CpuAES, + Modrm|IgnoreSize|NoSuf, + { Imm8, RegXMM|LLongMem, + RegXMM } }, + + /* Intel Carry-less Multiplication extensions */ + {"pclmulqdq", 3, 0x660f3a44, None, CpuPCLMUL, + Modrm|IgnoreSize|NoSuf, + { Imm8, RegXMM|LLongMem, + RegXMM } }, + {"pclmullqlqdq", 2, 0x660f3a44, 0x0, CpuPCLMUL, + Modrm|IgnoreSize|NoSuf|ImmExt, + { RegXMM|LLongMem, + RegXMM } }, + {"pclmulhqlqdq", 2, 0x660f3a44, 0x1, CpuPCLMUL, + Modrm|IgnoreSize|NoSuf|ImmExt, + { RegXMM|LLongMem, + RegXMM } }, + {"pclmullqhqdq", 2, 0x660f3a44, 0x10, CpuPCLMUL, + Modrm|IgnoreSize|NoSuf|ImmExt, + { RegXMM|LLongMem, + RegXMM } }, + {"pclmulhqhqdq", 2, 0x660f3a44, 0x11, CpuPCLMUL, + Modrm|IgnoreSize|NoSuf|ImmExt, + { RegXMM|LLongMem, + RegXMM } }, + { NULL, 0, 0, 0, 0, 0, { 0 } } }; --IU5/I01NYhRvwH70-- From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 11:12:29 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 50C94424 for ; Thu, 17 Jan 2013 11:12:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id BC609368 for ; Thu, 17 Jan 2013 11:12:28 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0HBCOMN048618; Thu, 17 Jan 2013 13:12:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0HBCOMN048618 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0HBCOlU048617; Thu, 17 Jan 2013 13:12:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Jan 2013 13:12:24 +0200 From: Konstantin Belousov To: John-Mark Gurney Subject: Re: patch to add aes and pclmulqdq instructions to gcc Message-ID: <20130117111224.GP2522@kib.kiev.ua> References: <20130117070516.GI1410@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5/6IVfYouxg+lu1D" Content-Disposition: inline In-Reply-To: <20130117070516.GI1410@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 11:12:29 -0000 --5/6IVfYouxg+lu1D Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 16, 2013 at 11:05:16PM -0800, John-Mark Gurney wrote: > Mike Belopuhov pointed me to the patch in OpenBSD: > http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef70462= 9e/diff.txt >=20 > While OpenBSD's binutils is quite different than FreeBSD's, I was able > to use his patch to teach binutils how to assemble and disassemble the > aes and pclmulqdq instructions. >=20 > I have done basic tests, such as verified that it can assemble the aesni > module and get the same results, and assemble a sample file for > pclmulqdq.. For each of these tests, I have verified that it's output > matches (as close as possible, as gcc/clang compile callq's differently) > clang on amd64.. Did you removed the manually assembled bytes from aes*.S and replaced them with the commented-out instructions for the test ? There is also newer VEX encoding for the same AESNI set, but adding the support for them might be hard because the source base is old. --5/6IVfYouxg+lu1D Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ99yXAAoJEJDCuSvBvK1BhFQP/RCxf03+i0N9hWQZHQFyAq+n 17lAoG0SGqGSNb29/bfINE6LZVlxAgpB8KnaHDQwMudBuJ7WEH6W0lALKrP7n4Wv bJ54Cn4rzDfpWuzxo3Gh847UZnNziwfVntplUq6T6Mpi8YEvkE4tbxR3ndDvqLIl hVvSntD3ZZ1Vy2gSvAaA2TDF/d1Rv2iAzCUTVfGAjlLu+/4SyuY8BMVhKAN6y5K4 JPhKopHiNgQEudcIi+EgvO9nRtX0zcLqymyvqAg7jwfYlN1I8xqxx8q1A1ABLHCe 6hGjiMxA9CbYy50hY57YJ7w4SKqVNDPWhVZ83/JotVRp45EIb2pf7n4DdKER+r17 Hq1Jrw9yvWNA5RAxBcnq5n/JORKxKewcxgf74Yf1gmVepG09i47KczjXoFK6wsii malBQw7PIKJ3Ug+eBzWMcyzhJ+5ioS8bUcEl5xpihYgmLLb/D1pUi8vM2YBGVmfL 4xwDYmKgyZMBhQKn5Dj02l/quEZRoUaoacb8g+pZ+tWLMoP0YU898c11QMBdye01 dvLFZtg5xUYmtFrEyG009M0PCg9tK3Q/GyK6wOABoHn2p6+zsxvIwgALJ37vX4zT 1BVe90xhFzifynT3fWqDYS1ArkmU8sqOpgwJBMuidaelcLVAKgEb2cU1WvlYEi4G bb8P8QZXBS/2djMdlkq4 =mfaX -----END PGP SIGNATURE----- --5/6IVfYouxg+lu1D-- From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 14:12:26 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C3C13DE3 for ; Thu, 17 Jan 2013 14:12:26 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f44.google.com (mail-pb0-f44.google.com [209.85.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id A1269ED for ; Thu, 17 Jan 2013 14:12:26 +0000 (UTC) Received: by mail-pb0-f44.google.com with SMTP id uo1so1425756pbc.3 for ; Thu, 17 Jan 2013 06:12:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=+9/9PrkPZmekRTZzNnTPZDfYBFVZrnG0pG3XBiq93Y8=; b=UlyGy2cAEvrGoSKfewaeb0L7sk3oe7UzqzvQEczHk/jM+S9lBAngwdjwAbvhRe2Y6C zFvn4idGWEYuBksMiVqM/Ngjusq5KCFV8vOuG5BgdA582HcYp9TWz1B8A7YJxO3wrins Cbd0j3M40p7oS9REcw3rjMUPOpxCdbHw/V64hl7ExkjGzNaQJ0sCiJvltX81HiUMBRsl xmKrX10E3S1dVHn3KWin7WYFCFZXmdP4bN23hW7F41AYZyiDaZDF0qs2bZK9HCwQjuqW ye8zQD3oHxRz+CeVZkighxpmGRS7U5jc8B5bobt4g5SF/tZk4yWNddHyATeP4bJfQZ1h Qycw== X-Received: by 10.68.239.104 with SMTP id vr8mr13842472pbc.59.1358431486504; Thu, 17 Jan 2013 06:04:46 -0800 (PST) Received: from [10.0.0.53] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id sb3sm1129866pbc.44.2013.01.17.06.04.44 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Jan 2013 06:04:44 -0800 (PST) Sender: Warner Losh Subject: Re: patch to add aes and pclmulqdq instructions to gcc Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <20130117070516.GI1410@funkthat.com> Date: Thu, 17 Jan 2013 07:04:40 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <75C84A9C-E12A-4D2E-8474-46678932236B@bsdimp.com> References: <20130117070516.GI1410@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQkZfX3YE0Z1EEQb+Tt0aCjDUzu5VYD4EmbRI+qIy0/QcfeaJsgW9StvEvUmpwnqv66BoGeC Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 14:12:26 -0000 On Jan 17, 2013, at 12:05 AM, John-Mark Gurney wrote: > Mike Belopuhov pointed me to the patch in OpenBSD: > = http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef704629= e/diff.txt >=20 > While OpenBSD's binutils is quite different than FreeBSD's, I was able > to use his patch to teach binutils how to assemble and disassemble the > aes and pclmulqdq instructions. >=20 > I have done basic tests, such as verified that it can assemble the = aesni > module and get the same results, and assemble a sample file for > pclmulqdq.. For each of these tests, I have verified that it's output > matches (as close as possible, as gcc/clang compile callq's = differently) > clang on amd64.. >=20 > I have attached the patch, and it is also availble at: > http://people.freebsd.org/~jmg/gcc.aes.patch >=20 > Comments? >=20 > I have not passed it through a make universe yet, but will before > committing... >=20 > I am also working on basic intrinsics header files for these = instructions > too... This looks like it should do the trick. Warner= From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 17:31:41 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7A98B8FB for ; Thu, 17 Jan 2013 17:31:41 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 267D33A4 for ; Thu, 17 Jan 2013 17:31:40 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0HHVepE080106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Jan 2013 09:31:40 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0HHVegn080105; Thu, 17 Jan 2013 09:31:40 -0800 (PST) (envelope-from jmg) Date: Thu, 17 Jan 2013 09:31:40 -0800 From: John-Mark Gurney To: Konstantin Belousov Subject: Re: patch to add aes and pclmulqdq instructions to gcc Message-ID: <20130117173140.GJ1410@funkthat.com> References: <20130117070516.GI1410@funkthat.com> <20130117111224.GP2522@kib.kiev.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130117111224.GP2522@kib.kiev.ua> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 17 Jan 2013 09:31:40 -0800 (PST) Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 17:31:41 -0000 Konstantin Belousov wrote this message on Thu, Jan 17, 2013 at 13:12 +0200: > On Wed, Jan 16, 2013 at 11:05:16PM -0800, John-Mark Gurney wrote: > > Mike Belopuhov pointed me to the patch in OpenBSD: > > http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef704629e/diff.txt > > > > While OpenBSD's binutils is quite different than FreeBSD's, I was able > > to use his patch to teach binutils how to assemble and disassemble the > > aes and pclmulqdq instructions. > > > > I have done basic tests, such as verified that it can assemble the aesni > > module and get the same results, and assemble a sample file for > > pclmulqdq.. For each of these tests, I have verified that it's output > > matches (as close as possible, as gcc/clang compile callq's differently) > > clang on amd64.. > Did you removed the manually assembled bytes from aes*.S and replaced them > with the commented-out instructions for the test ? Yep.. > There is also newer VEX encoding for the same AESNI set, but adding the > support for them might be hard because the source base is old. I don't see a use for them right now. And considering how anoying it was to add just these instructions.. :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 17:53:08 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 57342185 for ; Thu, 17 Jan 2013 17:53:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id EEC876FB for ; Thu, 17 Jan 2013 17:53:07 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0HHr2IK090869; Thu, 17 Jan 2013 19:53:02 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0HHr2IK090869 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0HHr24c090868; Thu, 17 Jan 2013 19:53:02 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Jan 2013 19:53:02 +0200 From: Konstantin Belousov To: John-Mark Gurney Subject: Re: patch to add aes and pclmulqdq instructions to gcc Message-ID: <20130117175302.GS2522@kib.kiev.ua> References: <20130117070516.GI1410@funkthat.com> <20130117111224.GP2522@kib.kiev.ua> <20130117173140.GJ1410@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aJ74fq0Y6SrIeKCM" Content-Disposition: inline In-Reply-To: <20130117173140.GJ1410@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 17:53:08 -0000 --aJ74fq0Y6SrIeKCM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 17, 2013 at 09:31:40AM -0800, John-Mark Gurney wrote: > Konstantin Belousov wrote this message on Thu, Jan 17, 2013 at 13:12 +020= 0: > > On Wed, Jan 16, 2013 at 11:05:16PM -0800, John-Mark Gurney wrote: > > > Mike Belopuhov pointed me to the patch in OpenBSD: > > > http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef7= 04629e/diff.txt > > >=20 > > > While OpenBSD's binutils is quite different than FreeBSD's, I was able > > > to use his patch to teach binutils how to assemble and disassemble the > > > aes and pclmulqdq instructions. > > >=20 > > > I have done basic tests, such as verified that it can assemble the ae= sni > > > module and get the same results, and assemble a sample file for > > > pclmulqdq.. For each of these tests, I have verified that it's output > > > matches (as close as possible, as gcc/clang compile callq's different= ly) > > > clang on amd64.. > > Did you removed the manually assembled bytes from aes*.S and replaced t= hem > > with the commented-out instructions for the test ? >=20 > Yep.. Thank you, I expect that you will commit the crypto/aes change which removes the .bytes, after the gas patch. >=20 > > There is also newer VEX encoding for the same AESNI set, but adding the > > support for them might be hard because the source base is old. >=20 > I don't see a use for them right now. And considering how anoying it > was to add just these instructions.. :) I am not asking you to do more work. It just my understanding that Intel wants everybody migrate to VEX encoding, where available. --aJ74fq0Y6SrIeKCM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ+Dp9AAoJEJDCuSvBvK1BZzMP/iHZAu/YqhBOofURiep2R992 Wdj68su0EI5Fi6OaNmEmC2yjkg45umuJpdk2r1iH33QqrLrhtSt715VI9knm8C/e x67DFC7uZZGdBp1c5t6XzaD0+1Ro4IdXDOe40azWTfE2e2e55KolAc0nGZK2/db6 hWsr4VQcBSG3yLf4wAHchl7woEGUBUXcvSjb2JQh7MK43Cb2FBcjtVl3vJeESy4V wuVQXPqryqJrSMKeO9cBGa0UH+rZL1n+CISwWKcijxt7Rf3r7gHeCj4yNgngt+hf 7aitqFcdDnwZKNAb1ZcKf29MEhlpN9yliQCr4qgeQmhLRHT1y4QyR7SNL0H6VidI oM4MTipQKlB5vvEiSCstatmfh9imDfae69fsPrtJCReWMuTghY3WxgXpVuZeKhrr vACTNtQ+lfyXdulVT1gm2Q7FDbstyYk/qbLxdwraIleenOpA6UzRIyErJe6CXmsD nPtZywzDjSQurII8yk+uEkWYZJkVoMPwpEV9eKz1WR3Yo4i61ruGwNQqMcStZlNz AymmiwjCnf2uPB8dP7ky+VsMZt/TLSnUWThGmFTwWFwlZX+p/FMhKzbuaWI1TAUX /S5EhM7A/0PmX3CePjMQbtQIeT8Raf0pK177sPr0jlQfPTpxQCUTU1e5gi9rZzCH QuoFfC0S+ztZ8wM1uweh =Pido -----END PGP SIGNATURE----- --aJ74fq0Y6SrIeKCM-- From owner-freebsd-toolchain@FreeBSD.ORG Thu Jan 17 18:33:05 2013 Return-Path: Delivered-To: toolchain@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 490771B5 for ; Thu, 17 Jan 2013 18:33:05 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 107818D1 for ; Thu, 17 Jan 2013 18:33:04 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0HIX4Og081075 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Jan 2013 10:33:04 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0HIX42Q081074; Thu, 17 Jan 2013 10:33:04 -0800 (PST) (envelope-from jmg) Date: Thu, 17 Jan 2013 10:33:04 -0800 From: John-Mark Gurney To: Konstantin Belousov Subject: Re: patch to add aes and pclmulqdq instructions to gcc Message-ID: <20130117183304.GN1410@funkthat.com> References: <20130117070516.GI1410@funkthat.com> <20130117111224.GP2522@kib.kiev.ua> <20130117173140.GJ1410@funkthat.com> <20130117175302.GS2522@kib.kiev.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130117175302.GS2522@kib.kiev.ua> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 17 Jan 2013 10:33:04 -0800 (PST) Cc: toolchain@FreeBSD.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 18:33:05 -0000 Konstantin Belousov wrote this message on Thu, Jan 17, 2013 at 19:53 +0200: > On Thu, Jan 17, 2013 at 09:31:40AM -0800, John-Mark Gurney wrote: > > Konstantin Belousov wrote this message on Thu, Jan 17, 2013 at 13:12 +0200: > > > On Wed, Jan 16, 2013 at 11:05:16PM -0800, John-Mark Gurney wrote: > > > > Mike Belopuhov pointed me to the patch in OpenBSD: > > > > http://freshbsd.org/commit/openbsd/0babc91a00b1f1953637bb39c8ec97aef704629e/diff.txt > > > > > > > > While OpenBSD's binutils is quite different than FreeBSD's, I was able > > > > to use his patch to teach binutils how to assemble and disassemble the > > > > aes and pclmulqdq instructions. > > > > > > > > I have done basic tests, such as verified that it can assemble the aesni > > > > module and get the same results, and assemble a sample file for > > > > pclmulqdq.. For each of these tests, I have verified that it's output > > > > matches (as close as possible, as gcc/clang compile callq's differently) > > > > clang on amd64.. > > > Did you removed the manually assembled bytes from aes*.S and replaced them > > > with the commented-out instructions for the test ? > > > > Yep.. > Thank you, I expect that you will commit the crypto/aes change which > removes the .bytes, after the gas patch. Yep... Though I hope it'll be followed shortly by a much improved version of the AES-NI code.. :) Oh, does anyone see any problems w/ MFC'ing the patch(s)? I haven't done any testing yet (but I will), but from my understanding our gcc in 9 is very similar to 10? This is my first real foray into the toolchain like this... > > > There is also newer VEX encoding for the same AESNI set, but adding the > > > support for them might be hard because the source base is old. > > > > I don't see a use for them right now. And considering how anoying it > > was to add just these instructions.. :) > > I am not asking you to do more work. It just my understanding that > Intel wants everybody migrate to VEX encoding, where available. :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."