From owner-freebsd-toolchain@FreeBSD.ORG Sun Jan 6 18:02:30 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 733A4638; Sun, 6 Jan 2013 18:02:30 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from adsum.doit.wisc.edu (adsum.doit.wisc.edu [144.92.197.210]) by mx1.freebsd.org (Postfix) with ESMTP id A08AA10F7; Sun, 6 Jan 2013 18:02:30 +0000 (UTC) MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_hv2Jca5yydhH8mRnv/48vw)" Received: from avs-daemon.smtpauth1.wiscmail.wisc.edu by smtpauth1.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) id <0MG700100U3ZMM00@smtpauth1.wiscmail.wisc.edu>; Sun, 06 Jan 2013 12:02:23 -0600 (CST) Received: from wanderer.tachypleus.net (dhcp107-17-54-205.hil-sfofhhh.sfo.wayport.net [107.17.54.205]) by smtpauth1.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) with ESMTPSA id <0MG700M2DU3XBW10@smtpauth1.wiscmail.wisc.edu>; Sun, 06 Jan 2013 12:02:22 -0600 (CST) Date: Sun, 06 Jan 2013 10:02:21 -0800 From: Nathan Whitehorn Subject: LLVM Image Activator Sender: whitehorn@wisc.edu To: freebsd-arch@freebsd.org, freebsd-toolchain@freebsd.org Message-id: <50E9BC2D.7000302@freebsd.org> X-Spam-Report: AuthenticatedSender=yes, SenderIP=107.17.54.205 X-Spam-PmxInfo: Server=avs-14, Version=5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2013.1.6.175417, SenderIP=107.17.54.205 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2013 18:02:30 -0000 This is a multi-part message in MIME format. --Boundary_(ID_hv2Jca5yydhH8mRnv/48vw) Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Having LLVM/clang in the base system lets us do some interesting things that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM IR as well as components of a toolchain for it (this is what Google's pNACL uses) and that you can end up producing binary files that are in IR instead of native code. The IR isn't really cross-platform, but does let you do CPU-specific optimizations when executed by the JIT, etc. The attached patch causes the LLVM JIT (lli) to be built by default (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a kernel image activator that invokes it when passed LLVM bitcode files. It's not completely finished (see the XXX comment in the middle), but it does work, as follows: $ clang -emit-llvm -c -o hw.ll hw.c $ file hw.ll hw.ll: LLVM bitcode $ lli hw.ll Hello world! $ chmod a+x hw.ll $ ./hw.ll Hello world! $ Is there any interest in having features like this? It seems like this could provides some interesting possibilities for us and nice integration from having imported clang into base. -Nathan --Boundary_(ID_hv2Jca5yydhH8mRnv/48vw) Content-type: text/plain; CHARSET=US-ASCII; name=imgact_llvm.diff Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=imgact_llvm.diff Index: sys/kern/imgact_llvm.c =================================================================== --- sys/kern/imgact_llvm.c (revision 0) +++ sys/kern/imgact_llvm.c (working copy) @@ -0,0 +1,78 @@ +/*- + * Copyright (c) 2012 Nathan Whitehorn + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static const char llvm_magic[5] = {'B', 'C', 0xc0, 0xde, 0}; + +static int +exec_llvm_imgact(struct image_params *imgp) +{ + const char *interp = "/usr/bin/lli"; + + /* check for LLVM bitcode magic */ + if (memcmp(imgp->image_header, llvm_magic, 4) != 0) + return (-1); + + if (imgp->args->stringspace < strlen(interp) + 1) + return (ENOMEM); + + imgp->interpreted = 1; + + /* Move over arguments and set argv[0] to the interpreter */ + memmove(imgp->args->begin_argv + strlen(interp) + 1, + imgp->args->begin_argv, imgp->args->endp - imgp->args->begin_argv); + strcpy(imgp->args->begin_argv, interp); + + /* XXX: set argv[1] to fname instead of old argv[0], call -fake-argv0 with old argv[0] */ + + imgp->args->begin_envv += strlen(interp) + 1; + imgp->args->endp += strlen(interp) + 1; + imgp->args->stringspace -= strlen(interp) + 1; + imgp->args->argc++; + + imgp->interpreter_name = imgp->args->begin_argv; + + return (0); +} + +/* + * Tell kern_execve.c about it, with a little help from the linker. + */ +static struct execsw llvm_execsw = { exec_llvm_imgact, llvm_magic }; +EXEC_SET(llvm, llvm_execsw); + Property changes on: sys/kern/imgact_llvm.c ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: sys/conf/files =================================================================== --- sys/conf/files (revision 245062) +++ sys/conf/files (working copy) @@ -2586,6 +2586,7 @@ kern/device_if.m standard kern/imgact_elf.c standard kern/imgact_elf32.c optional compat_freebsd32 +kern/imgact_llvm.c standard kern/imgact_shell.c standard kern/inflate.c optional gzip kern/init_main.c standard Index: usr.bin/clang/Makefile =================================================================== --- usr.bin/clang/Makefile (revision 245062) +++ usr.bin/clang/Makefile (working copy) @@ -2,12 +2,11 @@ .include -SUBDIR= clang clang-tblgen tblgen +SUBDIR= clang clang-tblgen tblgen lli .if ${MK_CLANG_EXTRAS} != "no" && !defined(TOOLS_PREFIX) SUBDIR+=bugpoint \ llc \ - lli \ llvm-ar \ llvm-as \ llvm-bcanalyzer \ Index: lib/clang/Makefile =================================================================== --- lib/clang/Makefile (revision 245062) +++ lib/clang/Makefile (working copy) @@ -30,16 +30,23 @@ libllvmbitwriter \ libllvmcodegen \ libllvmcore \ + libllvmdebuginfo \ + libllvmexecutionengine \ libllvminstcombine \ libllvminstrumentation \ + libllvminterpreter \ libllvmipa \ libllvmipo \ + libllvmjit \ libllvmlinker \ libllvmmc \ + libllvmmcdisassembler \ + libllvmmcjit \ libllvmmcparser \ libllvmobject \ libllvmscalaropts \ libllvmselectiondag \ + libllvmruntimedyld \ libllvmsupport \ libllvmtablegen \ libllvmtarget \ @@ -68,18 +75,8 @@ libllvmx86disassembler \ libllvmx86info \ libllvmx86instprinter \ - libllvmx86utils - -.if ${MK_CLANG_EXTRAS} != "no" -SUBDIR+=libllvmdebuginfo \ - libllvmexecutionengine \ - libllvminterpreter \ - libllvmjit \ - libllvmmcdisassembler \ - libllvmmcjit \ - libllvmruntimedyld + libllvmx86utils .endif -.endif SUBDIR+= include Index: lib/clang/libllvmsupport/Makefile =================================================================== --- lib/clang/libllvmsupport/Makefile (revision 245062) +++ lib/clang/libllvmsupport/Makefile (working copy) @@ -16,12 +16,16 @@ ConstantRange.cpp \ CrashRecoveryContext.cpp \ DAGDeltaAlgorithm.cpp \ + DataExtractor.cpp \ + DataStream.cpp \ Debug.cpp \ DeltaAlgorithm.cpp \ + Disassembler.cpp \ Dwarf.cpp \ DynamicLibrary.cpp \ Errno.cpp \ ErrorHandling.cpp \ + FileUtilities.cpp \ FoldingSet.cpp \ FormattedStream.cpp \ GraphWriter.cpp \ @@ -56,6 +60,7 @@ StringMap.cpp \ StringPool.cpp \ StringRef.cpp \ + SystemUtils.cpp \ TargetRegistry.cpp \ ThreadLocal.cpp \ Threading.cpp \ @@ -76,15 +81,4 @@ system_error.cpp LLVM_REQUIRES_RTTI= -.if ${MK_CLANG_EXTRAS} != "no" -SRCS+= BlockFrequency.cpp \ - BranchProbability.cpp \ - DataExtractor.cpp \ - DataStream.cpp \ - Disassembler.cpp \ - FileUtilities.cpp \ - MemoryObject.cpp \ - SystemUtils.cpp -.endif - .include "../clang.lib.mk" --Boundary_(ID_hv2Jca5yydhH8mRnv/48vw)-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 09:18:03 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A8042AD8; Mon, 7 Jan 2013 09:18:03 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 6BFA8396; Mon, 7 Jan 2013 09:18:03 +0000 (UTC) Received: from JRE-MBP-2.local (c-50-143-148-105.hsd1.ca.comcast.net [50.143.148.105]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id r079I1BG066933 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 7 Jan 2013 01:18:02 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <50EA92C4.2080808@freebsd.org> Date: Mon, 07 Jan 2013 01:17:56 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Nathan Whitehorn Subject: Re: LLVM Image Activator References: <50E9BC2D.7000302@freebsd.org> In-Reply-To: <50E9BC2D.7000302@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 09:18:03 -0000 On 1/6/13 10:02 AM, Nathan Whitehorn wrote: > Having LLVM/clang in the base system lets us do some interesting things > that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM > IR as well as components of a toolchain for it (this is what Google's > pNACL uses) and that you can end up producing binary files that are in > IR instead of native code. The IR isn't really cross-platform, but does > let you do CPU-specific optimizations when executed by the JIT, etc. > > The attached patch causes the LLVM JIT (lli) to be built by default > (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a > kernel image activator that invokes it when passed LLVM bitcode files. > It's not completely finished (see the XXX comment in the middle), but it > does work, as follows: > > $ clang -emit-llvm -c -o hw.ll hw.c > $ file hw.ll > hw.ll: LLVM bitcode > $ lli hw.ll > Hello world! > $ chmod a+x hw.ll > $ ./hw.ll > Hello world! > $ > > Is there any interest in having features like this? It seems like this > could provides some interesting possibilities for us and nice > integration from having imported clang into base. > -Nathan sounds like a nice idea. I'd put it in if you can find a few other supporters.. (especially ones who can help with maintenance. > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 15:14:58 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 22614505; Mon, 7 Jan 2013 15:14:58 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 01D4BFB5; Mon, 7 Jan 2013 15:14:58 +0000 (UTC) Received: from ralph.baldwin.cx (c-68-39-198-164.hsd1.de.comcast.net [68.39.198.164]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 54263B984; Mon, 7 Jan 2013 10:14:57 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: LLVM Image Activator Date: Mon, 7 Jan 2013 09:36:38 -0500 User-Agent: KMail/1.13.7 (FreeBSD/9.1-PRERELEASE; KDE/4.8.4; amd64; ; ) References: <50E9BC2D.7000302@freebsd.org> In-Reply-To: <50E9BC2D.7000302@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201301070936.39052.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 07 Jan 2013 10:14:57 -0500 (EST) Cc: freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 15:14:58 -0000 On Sunday, January 06, 2013 01:02:21 PM Nathan Whitehorn wrote: > Having LLVM/clang in the base system lets us do some interesting things > that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM > IR as well as components of a toolchain for it (this is what Google's > pNACL uses) and that you can end up producing binary files that are in > IR instead of native code. The IR isn't really cross-platform, but does > let you do CPU-specific optimizations when executed by the JIT, etc. > > The attached patch causes the LLVM JIT (lli) to be built by default > (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a > kernel image activator that invokes it when passed LLVM bitcode files. > It's not completely finished (see the XXX comment in the middle), but it > does work, as follows: > > $ clang -emit-llvm -c -o hw.ll hw.c > $ file hw.ll > hw.ll: LLVM bitcode > $ lli hw.ll > Hello world! > $ chmod a+x hw.ll > $ ./hw.ll > Hello world! > $ > > Is there any interest in having features like this? It seems like this > could provides some interesting possibilities for us and nice > integration from having imported clang into base. > -Nathan This sounds neat indeed. Does the IR format provide any sort of notation for encoding the path to the interpreter (similar to ELF)? If not, you might want to at least make the path to 'lli' be configurable via a tunable and/or sysctl (e.g. if using a newer version of clang in /usr/local). -- John Baldwin From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 16:31:52 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B34FB809; Mon, 7 Jan 2013 16:31:52 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from mail-wg0-f45.google.com (mail-wg0-f45.google.com [74.125.82.45]) by mx1.freebsd.org (Postfix) with ESMTP id F0308633; Mon, 7 Jan 2013 16:31:51 +0000 (UTC) Received: by mail-wg0-f45.google.com with SMTP id dq12so10159175wgb.24 for ; Mon, 07 Jan 2013 08:31:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:from:to:subject:date:user-agent:cc:references :in-reply-to:mime-version:content-type:content-transfer-encoding :message-id; bh=kUMribF3yu7gycPE1tU3/fD0BBWKU6s4mjyhSWIzMrU=; b=ez2P/4Hw8zHwTfArti+Q2QgWVrL6c0JajtEm9qKmiatKFyk9/XOyJf35hWRgv63H9v lXC1dKyGG7jh0h0myLoCEgEagJDfZNHXSSdkTcY6RufqdFhoi2MGhqe8dhcyZzglP5v3 khakCD170AcBrTVUAnQ7u4i15hfntSlKMwe4mgyKFZc0c2Z6U7spW2h1LR8jBOOFqwb3 EO3D7rjb3YsGwTuLuldc4owkOgGnh0x2lWDlBQ2l6Fiz+iDZ01IZ7KVK+zKc6SoUFst8 JujnICAlD4pYhVzfFi0d85gRAAnqLT22z9AVez9cWoBzFDy3mYXKx3D4tzANB5tEUob6 YWtA== X-Received: by 10.180.78.137 with SMTP id b9mr10112881wix.30.1357575912696; Mon, 07 Jan 2013 08:25:12 -0800 (PST) Received: from dragon.dg (41-135-148-131.dsl.mweb.co.za. [41.135.148.131]) by mx.google.com with ESMTPS id h19sm12811757wiv.7.2013.01.07.08.25.09 (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 07 Jan 2013 08:25:11 -0800 (PST) From: David Naylor To: freebsd-toolchain@freebsd.org Subject: Re: LLVM Image Activator Date: Mon, 7 Jan 2013 18:25:03 +0200 User-Agent: KMail/1.13.7 (FreeBSD/9.1-PRERELEASE; KDE/4.9.1; amd64; ; ) References: <50E9BC2D.7000302@freebsd.org> In-Reply-To: <50E9BC2D.7000302@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1362420.XBIvH0nEZn"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201301071825.06439.naylor.b.david@gmail.com> Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 16:31:52 -0000 --nextPart1362420.XBIvH0nEZn Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Just my 2c On Sunday, 6 January 2013 20:02:21 Nathan Whitehorn wrote: > Having LLVM/clang in the base system lets us do some interesting things > that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM > IR as well as components of a toolchain for it (this is what Google's > pNACL uses) and that you can end up producing binary files that are in > IR instead of native code. The IR isn't really cross-platform, but does > let you do CPU-specific optimizations when executed by the JIT, etc. >=20 > The attached patch causes the LLVM JIT (lli) to be built by default > (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a > kernel image activator that invokes it when passed LLVM bitcode files. > It's not completely finished (see the XXX comment in the middle), but it > does work, as follows: >=20 > $ clang -emit-llvm -c -o hw.ll hw.c > $ file hw.ll > hw.ll: LLVM bitcode > $ lli hw.ll > Hello world! > $ chmod a+x hw.ll > $ ./hw.ll > Hello world! > $ >=20 > Is there any interest in having features like this? It seems like this > could provides some interesting possibilities for us and nice > integration from having imported clang into base. Would it be possible to have this as a module (and thus in ports)? Or,=20 perhaps, change the sources such these things could be loaded as a module... Also, with modification to LLVM, wouldn't the existing '#!' mechanism work= =20 (aka '#!/usr/bin/lli') at the beginning of the file? =20 On an aside. Could you imagine shipping a 'x86' llvm-byte code and get the= =20 jit (with caching), thus your computer will get full support (speed) from t= he=20 binaries and the binaries will work on all 'x86' related architectures. (M= y=20 thinking is for things like i386 vs pentium4). =20 Regards --nextPart1362420.XBIvH0nEZn Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEABECAAYFAlDq9uIACgkQUaaFgP9pFrKo3ACeNX5ZhxuslKEem1bHkRpR8YQd HDoAn1Ukz4GTpxieoB7u1C7UFCSJsV90 =r9wK -----END PGP SIGNATURE----- --nextPart1362420.XBIvH0nEZn-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 16:46:00 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0045BC34; Mon, 7 Jan 2013 16:45:59 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from agogare.doit.wisc.edu (agogare.doit.wisc.edu [144.92.197.211]) by mx1.freebsd.org (Postfix) with ESMTP id BFC7C6EA; Mon, 7 Jan 2013 16:45:59 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) id <0MG900G02L7Z7X00@smtpauth2.wiscmail.wisc.edu>; Mon, 07 Jan 2013 10:45:35 -0600 (CST) Received: from wanderer.tachypleus.net (216-75-226-134.static.wiline.com [216.75.226.134]) by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) with ESMTPSA id <0MG900BB1L7WDD30@smtpauth2.wiscmail.wisc.edu>; Mon, 07 Jan 2013 10:45:33 -0600 (CST) Date: Mon, 07 Jan 2013 08:45:32 -0800 From: Nathan Whitehorn Subject: Re: LLVM Image Activator In-reply-to: <201301071825.06439.naylor.b.david@gmail.com> Sender: whitehorn@wisc.edu To: David Naylor Message-id: <50EAFBAC.2020808@freebsd.org> X-Spam-Report: AuthenticatedSender=yes, SenderIP=216.75.226.134 X-Spam-PmxInfo: Server=avs-14, Version=5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2013.1.7.163317, SenderIP=216.75.226.134 References: <50E9BC2D.7000302@freebsd.org> <201301071825.06439.naylor.b.david@gmail.com> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 16:46:00 -0000 On 01/07/13 08:25, David Naylor wrote: > Hi, > > Just my 2c > > On Sunday, 6 January 2013 20:02:21 Nathan Whitehorn wrote: >> Having LLVM/clang in the base system lets us do some interesting things >> that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM >> IR as well as components of a toolchain for it (this is what Google's >> pNACL uses) and that you can end up producing binary files that are in >> IR instead of native code. The IR isn't really cross-platform, but does >> let you do CPU-specific optimizations when executed by the JIT, etc. >> >> The attached patch causes the LLVM JIT (lli) to be built by default >> (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a >> kernel image activator that invokes it when passed LLVM bitcode files. >> It's not completely finished (see the XXX comment in the middle), but it >> does work, as follows: >> >> $ clang -emit-llvm -c -o hw.ll hw.c >> $ file hw.ll >> hw.ll: LLVM bitcode >> $ lli hw.ll >> Hello world! >> $ chmod a+x hw.ll >> $ ./hw.ll >> Hello world! >> $ >> >> Is there any interest in having features like this? It seems like this >> could provides some interesting possibilities for us and nice >> integration from having imported clang into base. > > Would it be possible to have this as a module (and thus in ports)? Or, > perhaps, change the sources such these things could be loaded as a module... We do support that, but I'm not really sure what would be gained. We have LLVM in base; it seems a shame not to use it. > Also, with modification to LLVM, wouldn't the existing '#!' mechanism work > (aka '#!/usr/bin/lli') at the beginning of the file? In principle, yes, but we'd have to coordinate that with a lot of players since a number of tools operate on LLVM bitcode. The other obstacle is that LLVM bitcode is a giant binary blob, not text, so the standard #! seems a little out of place. > On an aside. Could you imagine shipping a 'x86' llvm-byte code and get the > jit (with caching), thus your computer will get full support (speed) from the > binaries and the binaries will work on all 'x86' related architectures. (My > thinking is for things like i386 vs pentium4). > Yes, this is exactly the kind of thing I had in mind. -Nathan From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 17:24:40 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5D1A18C4; Mon, 7 Jan 2013 17:24:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id AE1228E2; Mon, 7 Jan 2013 17:24:39 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r07HOXL4060397; Mon, 7 Jan 2013 19:24:33 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua r07HOXL4060397 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r07HOX9J060396; Mon, 7 Jan 2013 19:24:33 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 7 Jan 2013 19:24:33 +0200 From: Konstantin Belousov To: John Baldwin Subject: Re: LLVM Image Activator Message-ID: <20130107172433.GX82219@kib.kiev.ua> References: <50E9BC2D.7000302@freebsd.org> <201301070936.39052.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oDFa8iZvINJ+yMtC" Content-Disposition: inline In-Reply-To: <201301070936.39052.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 17:24:40 -0000 --oDFa8iZvINJ+yMtC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 07, 2013 at 09:36:38AM -0500, John Baldwin wrote: > On Sunday, January 06, 2013 01:02:21 PM Nathan Whitehorn wrote: > > Having LLVM/clang in the base system lets us do some interesting things > > that we couldn't do with GCC. One is that LLVM ships with a JIT for LLVM > > IR as well as components of a toolchain for it (this is what Google's > > pNACL uses) and that you can end up producing binary files that are in > > IR instead of native code. The IR isn't really cross-platform, but does > > let you do CPU-specific optimizations when executed by the JIT, etc. > >=20 > > The attached patch causes the LLVM JIT (lli) to be built by default > > (adding ~20 seconds to buildworld on my five-year-old laptop) and adds a > > kernel image activator that invokes it when passed LLVM bitcode files. > > It's not completely finished (see the XXX comment in the middle), but it > > does work, as follows: > >=20 > > $ clang -emit-llvm -c -o hw.ll hw.c > > $ file hw.ll > > hw.ll: LLVM bitcode > > $ lli hw.ll > > Hello world! > > $ chmod a+x hw.ll > > $ ./hw.ll > > Hello world! > > $ > >=20 > > Is there any interest in having features like this? It seems like this > > could provides some interesting possibilities for us and nice > > integration from having imported clang into base. > > -Nathan >=20 > This sounds neat indeed. Does the IR format provide any sort of notation= for=20 > encoding the path to the interpreter (similar to ELF)? If not, you might= want=20 > to at least make the path to 'lli' be configurable via a tunable and/or s= ysctl=20 > (e.g. if using a newer version of clang in /usr/local). I do not like one-purpose write-once never-used image activators. I still do remember the buzz about the binary format 0xCAFEBABE, which AFAIR gained image activator support on several OSes, to be garbage collected. For one thing, such hacks are very system-specific and thus are never used, causing quick bit rot. Second, more important, is that parsing of yet another format opens an attack vector on kernel, and due to the nature of the activators, on the user space. Look at the PE image activator history, including at least one SA, to see what I mean. And PE is real image format instead of hack. Even for ELF, there were recent panic-level bugs. Note that several language runtimes are happy to use the shebang for bytecode. For examples, look at the ocaml and lisps. If adding despite these arguments, please do not make the thing 'standard'. Provide the option to compile it in, and may be make it a module. --oDFa8iZvINJ+yMtC Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ6wTQAAoJEJDCuSvBvK1BM4sP/1zs6BH/0cXbfHukn+M0TOLz S5+Dn+FTd9K4FmXS2CQpQdcZDCE1uBTQ8j6pIyUCO4rOSaGmdYTCOij2TRnEs+Rf v1wdr49/H7iO1uDAG2l32WPSmvd2K6tFilDRiLvSsBaH63uI2+xGhq0jPfSXPbv+ jnsBDOGSoex2+rIaDvoV9TZ/2ShWEDp2SBnPPBYTa515JmO0ypw2FBZd5+9wbW7v oaLNhQQz7eB1nrlxCs2MlQB4PR94DH9xAmkdRp+Q/AJ58icOiZOBiEH2L9ESYF+n UFP32ESKJrABTvzoS+nusRtmQwWZrymFUpb1AxqXVbi8fm2/9dOf1aLqbH9dO2vT IiuzC/C14yvp01DSs66ryD7ODogEaTEIMbBx9CHOkmcka4vxxPhqhX3is7xMnnVs ipcb0DeKiv7FIYwinTkJzrXhS4gi5N6nR+1RqEsq2LZW6BywERhVR3S4wNgWJWJH VisAiQUrUlwxpEwm6OJaA+PQ1OKlx80ZRYVJwC9Wr6gml9TRqjlOvO7VLXx4+K1v 3T2u8ha0qPEApKvIM5s2Ws2Npj4qkAbjHl8hxfTOYe++HQgjbNYVnJnAZ+7nm0CN u9YsM61C+u9W5X3ROmXiKcqsZf2VSekJhQNXXebgqs5BB+rG+tiIWijuzst9IF5S 3gpSF0hDaZf3X4suEe6w =i6JE -----END PGP SIGNATURE----- --oDFa8iZvINJ+yMtC-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 18:22:48 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 83877917; Mon, 7 Jan 2013 18:22:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id C02D3B3D; Mon, 7 Jan 2013 18:22:47 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r07IMaZe066239; Mon, 7 Jan 2013 20:22:36 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua r07IMaZe066239 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r07IMZkb066238; Mon, 7 Jan 2013 20:22:35 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 7 Jan 2013 20:22:35 +0200 From: Konstantin Belousov To: arch@freebsd.org Subject: Fast sigblock (AKA rtld speedup) Message-ID: <20130107182235.GA65279@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="sdtB3X0nJg68CQEu" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 18:22:48 -0000 --sdtB3X0nJg68CQEu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Below is the forward of the patch for which I failed to obtain a private review. Might be, the list generates more responses. Our rtld has a performance bootleneck, typically exposed by the images with the lot of the run-time relocation processing, and by the C++ exception handling. We block the signals delivery during the rtld performing the relocations, as well as for the dl_iterate_phdr(3) (the later is used for finding the dwarf unwinding tables). The signal blocking is needed to allow the rtld to resolve the symbols for the signal handlers in the safe way, but also causes 2 syscalls overhead per each rtld entry. The proposed approach allows to shave off those two syscalls, doubling the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. Date: Mon, 13 Aug 2012 15:26:00 +0300 =46rom: Konstantin Belousov =2E.. The basic idea is to implement sigprocmask() as single write into usermode address. If kernel needs to calculate the signal mask for current thread, it takes into the consideration non-zero value of the word at the agreed address. Also, usermode is informed about signals which were put on hold due to fast sigblock active. As I said, on my measurements in microbenchmark that did throw/catch in a loop, I see equal user and system time spent for unpatched system, and same user time with zero system time on patched system. Patch can be improved further, e.g. it would be nice to allow rtld to fall back to sigprocmask(2) if kernel does not support fast sigblock, to prevent flag day. Also, the mask enforced by fast sigblock can be made configurable. Note that libthr already blocks signals by catching them, and not using rtld service in the first line handler. I tried to make the change in the spirit of libthr interceptors, but handoff to libthr appears too complicated to work. In fact, libthr can be changed to start using fast sigblock instead of wrapping sigaction, but this is out of scope of the proposal right now. Please comment. diff --git a/lib/libc/sys/Symbol.map b/lib/libc/sys/Symbol.map index 6888ea0..3b75539 100644 --- a/lib/libc/sys/Symbol.map +++ b/lib/libc/sys/Symbol.map @@ -546,6 +546,7 @@ FBSDprivate_1.0 { __sys_extattr_set_link; _extattrctl; __sys_extattrctl; + __sys_fast_sigblock; _fchdir; __sys_fchdir; _fchflags; diff --git a/libexec/rtld-elf/rtld_lock.c b/libexec/rtld-elf/rtld_lock.c index d1563e5..50c52c6 100644 --- a/libexec/rtld-elf/rtld_lock.c +++ b/libexec/rtld-elf/rtld_lock.c @@ -43,6 +43,7 @@ */ =20 #include +#include #include #include #include @@ -59,8 +60,8 @@ typedef struct Struct_Lock { void *base; } Lock; =20 -static sigset_t fullsigmask, oldsigmask; static int thread_flag; +static uint32_t fsigblock; =20 static void * def_lock_create() @@ -111,18 +112,26 @@ def_rlock_acquire(void *lock) } =20 static void +sig_fastunblock(void) +{ + uint32_t oldval; + + oldval =3D atomic_fetchadd_32(&fsigblock, -FAST_SIGBLOCK_INC); + if (oldval =3D=3D (FAST_SIGBLOCK_PEND | FAST_SIGBLOCK_INC)) + __sys_fast_sigblock(FAST_SIGBLOCK_UNBLOCK, NULL); +} + +static void def_wlock_acquire(void *lock) { Lock *l =3D (Lock *)lock; - sigset_t tmp_oldsigmask; =20 for ( ; ; ) { - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); + atomic_add_rel_32(&fsigblock, FAST_SIGBLOCK_INC); if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) break; - sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); + sig_fastunblock(); } - oldsigmask =3D tmp_oldsigmask; } =20 static void @@ -134,7 +143,7 @@ def_lock_release(void *lock) atomic_add_rel_int(&l->lock, -RC_INCR); else { atomic_add_rel_int(&l->lock, -WAFLAG); - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); + sig_fastunblock(); } } =20 @@ -286,19 +295,7 @@ lockdflt_init() =20 memcpy(&lockinfo, &deflockinfo, sizeof(lockinfo)); _rtld_thread_init(NULL); - /* - * Construct a mask to block all signals except traps which might - * conceivably be generated within the dynamic linker itself. - */ - sigfillset(&fullsigmask); - sigdelset(&fullsigmask, SIGILL); - sigdelset(&fullsigmask, SIGTRAP); - sigdelset(&fullsigmask, SIGABRT); - sigdelset(&fullsigmask, SIGEMT); - sigdelset(&fullsigmask, SIGFPE); - sigdelset(&fullsigmask, SIGBUS); - sigdelset(&fullsigmask, SIGSEGV); - sigdelset(&fullsigmask, SIGSYS); + __sys_fast_sigblock(FAST_SIGBLOCK_SETPTR, &fsigblock); } =20 /* @@ -319,7 +316,10 @@ _rtld_thread_init(struct RtldLockInfo *pli) =20 if (pli =3D=3D NULL) pli =3D &deflockinfo; - + else { + fsigblock =3D 0; + __sys_fast_sigblock(FAST_SIGBLOCK_UNSETPTR, NULL); + } =20 for (i =3D 0; i < RTLD_LOCK_CNT; i++) if ((locks[i] =3D pli->lock_create()) =3D=3D NULL) diff --git a/sys/compat/freebsd32/syscalls.master b/sys/compat/freebsd32/sy= scalls.master index 0891e41..f9e8b9e 100644 --- a/sys/compat/freebsd32/syscalls.master +++ b/sys/compat/freebsd32/syscalls.master @@ -997,3 +997,4 @@ uint32_t offset1, uint32_t offset2,\ uint32_t len1, uint32_t len2, \ int advice); } +532 AUE_NULL NOPROTO { int fast_sigblock(int cmd, uint32_t *ptr); } diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c index 90f7311..8a3cd15 100644 --- a/sys/kern/kern_exec.c +++ b/sys/kern/kern_exec.c @@ -1031,6 +1031,7 @@ exec_new_vmspace(imgp, sv) int error; struct proc *p =3D imgp->proc; struct vmspace *vmspace =3D p->p_vmspace; + struct thread *td =3D curthread; vm_object_t obj; vm_offset_t sv_minuser, stack_addr; vm_map_t map; @@ -1039,6 +1040,10 @@ exec_new_vmspace(imgp, sv) imgp->vmspace_destroyed =3D 1; imgp->sysent =3D sv; =20 + td->td_pflags &=3D ~TDP_FAST_SIGBLOCK; + td->td_sigblock_ptr =3D NULL; + td->td_sigblock_val =3D 0; + /* May be called with Giant held */ EVENTHANDLER_INVOKE(process_exec, p, imgp); =20 diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 2685a8b..3c8a2f7 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -230,6 +230,7 @@ static int sigproptbl[NSIG] =3D { }; =20 static void reschedule_signals(struct proc *p, sigset_t block, int flags); +static sigset_t fastblock_mask; =20 static void sigqueue_start(void) @@ -240,6 +241,16 @@ sigqueue_start(void) p31b_setcfg(CTL_P1003_1B_REALTIME_SIGNALS, _POSIX_REALTIME_SIGNALS); p31b_setcfg(CTL_P1003_1B_RTSIG_MAX, SIGRTMAX - SIGRTMIN + 1); p31b_setcfg(CTL_P1003_1B_SIGQUEUE_MAX, max_pending_per_proc); + SIGFILLSET(fastblock_mask); + SIG_CANTMASK(fastblock_mask); + SIGDELSET(fastblock_mask, SIGILL); + SIGDELSET(fastblock_mask, SIGTRAP); + SIGDELSET(fastblock_mask, SIGABRT); + SIGDELSET(fastblock_mask, SIGEMT); + SIGDELSET(fastblock_mask, SIGFPE); + SIGDELSET(fastblock_mask, SIGBUS); + SIGDELSET(fastblock_mask, SIGSEGV); + SIGDELSET(fastblock_mask, SIGSYS); } =20 ksiginfo_t * @@ -2525,6 +2536,7 @@ issignal(struct thread *td, int stop_allowed) struct sigqueue *queue; sigset_t sigpending; int sig, prop, newsig; + uint32_t oldval; =20 p =3D td->td_proc; ps =3D p->p_sigacts; @@ -2541,6 +2553,32 @@ issignal(struct thread *td, int stop_allowed) SIG_STOPSIGMASK(sigpending); if (SIGISEMPTY(sigpending)) /* no signal to send */ return (0); + + /* + * Do fast sigblock if requested by usermode. Since + * we do know that there was a signal pending at this + * point, set the FAST_SIGBLOCK_PEND as indicator for + * usermode to perform a dummy call to + * FAST_SIGBLOCK_UNBLOCK, which causes immediate + * delivery of postponed pending signal. + */ + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { + if (td->td_sigblock_val !=3D 0) + SIGSETNAND(sigpending, fastblock_mask); + if (SIGISEMPTY(sigpending)) { + oldval =3D fuword32(td->td_sigblock_ptr); + if (oldval =3D=3D -1) { + fetch_fast_sigblock_failed(td, 0); + return (0); + } + oldval |=3D FAST_SIGBLOCK_PEND; + if (suword32(td->td_sigblock_ptr, oldval) =3D=3D -1) + fetch_fast_sigblock_failed(td, 1); + td->td_sigblock_val =3D oldval; + return (0); + } + } + sig =3D sig_ffs(&sigpending); =20 if (p->p_stops & S_SIG) { @@ -3456,3 +3494,92 @@ sigacts_shared(struct sigacts *ps) mtx_unlock(&ps->ps_mtx); return (shared); } + +int +sys_fast_sigblock(struct thread *td, struct fast_sigblock_args *uap) +{ + struct proc *p; + int error; + uint32_t oldval; + + error =3D 0; + switch (uap->cmd) { + case FAST_SIGBLOCK_SETPTR: + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) + error =3D EBUSY; + else if (((uintptr_t)(uap->ptr) & (sizeof(uint32_t) - 1)) !=3D 0) + error =3D EINVAL; + else { + td->td_pflags |=3D TDP_FAST_SIGBLOCK; + td->td_sigblock_ptr =3D uap->ptr; + } + break; + + case FAST_SIGBLOCK_UNBLOCK: + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { + oldval =3D casuword32(td->td_sigblock_ptr, + FAST_SIGBLOCK_PEND, 0); + if (oldval =3D=3D (uint32_t)-1) + error =3D EFAULT; + else if (oldval !=3D FAST_SIGBLOCK_PEND) + error =3D EBUSY; + else + td->td_sigblock_val =3D 0; + } else + error =3D EDOOFUS; + + /* + * Rely on normal ast mechanism to deliver pending + * signals to current thread. But notify others about + * fake unblock. + */ + p =3D td->td_proc; + if (error =3D=3D 0 && p->p_numthreads !=3D 1) { + PROC_LOCK(p); + reschedule_signals(p, td->td_sigmask, 0); + PROC_UNLOCK(p); + } + break; + + case FAST_SIGBLOCK_UNSETPTR: + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { + error =3D copyin(td->td_sigblock_ptr, &oldval, + sizeof(uint32_t)); + if (error !=3D 0) + break; + if (oldval !=3D 0 && oldval !=3D FAST_SIGBLOCK_PEND) + error =3D EBUSY; + else { + td->td_pflags &=3D ~TDP_FAST_SIGBLOCK; + td->td_sigblock_val =3D 0; + } + } else + error =3D EDOOFUS; + break; + } + return (error); +} + +void +fetch_fast_sigblock(struct thread *td) +{ + + if ((td->td_pflags & TDP_FAST_SIGBLOCK) =3D=3D 0) + return; + td->td_sigblock_val =3D fuword32(td->td_sigblock_ptr); + if (td->td_sigblock_val =3D=3D -1) + fetch_fast_sigblock_failed(td, 0); +} + +void +fetch_fast_sigblock_failed(struct thread *td, int write) +{ + ksiginfo_t ksi; + + td->td_sigblock_val =3D 0; + ksiginfo_init_trap(&ksi); + ksi.ksi_signo =3D SIGSEGV; + ksi.ksi_code =3D write ? SEGV_ACCERR : SEGV_MAPERR; + ksi.ksi_addr =3D td->td_sigblock_ptr; + trapsignal(td, &ksi); +} diff --git a/sys/kern/subr_syscall.c b/sys/kern/subr_syscall.c index 5aee684..77b250d 100644 --- a/sys/kern/subr_syscall.c +++ b/sys/kern/subr_syscall.c @@ -131,6 +131,12 @@ syscallenter(struct thread *td, struct syscall_args *s= a) sa->callp, sa->args, 0); #endif =20 + /* + * Fetch fast sigblock value at the time of syscall + * entry because sleepqueue primitives might call + * cursig(). + */ + fetch_fast_sigblock(td); AUDIT_SYSCALL_ENTER(sa->code, td); error =3D (sa->callp->sy_call)(td, sa->args); AUDIT_SYSCALL_EXIT(error, td); diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c index 095bbdf..66e485b 100644 --- a/sys/kern/subr_trap.c +++ b/sys/kern/subr_trap.c @@ -237,6 +237,7 @@ ast(struct trapframe *framep) */ if (flags & TDF_NEEDSIGCHK || p->p_pendingcnt > 0 || !SIGISEMPTY(p->p_siglist)) { + fetch_fast_sigblock(td); PROC_LOCK(p); mtx_lock(&p->p_sigacts->ps_mtx); while ((sig =3D cursig(td, SIG_STOP_ALLOWED)) !=3D 0) diff --git a/sys/kern/syscalls.master b/sys/kern/syscalls.master index f62dad7..28a9393 100644 --- a/sys/kern/syscalls.master +++ b/sys/kern/syscalls.master @@ -951,5 +951,6 @@ off_t offset, off_t len); } 531 AUE_NULL STD { int posix_fadvise(int fd, off_t offset, \ off_t len, int advice); } +532 AUE_NULL STD { int fast_sigblock(int cmd, uint32_t *ptr); } ; Please copy any additions and changes to the following compatability tab= les: ; sys/compat/freebsd32/syscalls.master diff --git a/sys/sys/proc.h b/sys/sys/proc.h index 06df632..4899ca2 100644 --- a/sys/sys/proc.h +++ b/sys/sys/proc.h @@ -272,6 +272,9 @@ struct thread { struct osd td_osd; /* (k) Object specific data. */ struct vm_map_entry *td_map_def_user; /* (k) Deferred entries. */ pid_t td_dbg_forked; /* (c) Child pid for debugger. */ + void *td_sigblock_ptr; /* (k) uptr for fast sigblock. */ + uint32_t td_sigblock_val; /* (k) fast sigblock value at + kernel entry. */ #define td_endzero td_sigmask =20 /* Copied during fork1() or create_thread(). */ @@ -424,6 +427,7 @@ do { \ #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */ #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ #define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */ +#define TDP_FAST_SIGBLOCK 0x20000000 /* Fast sigblock active */ =20 /* * Reasons that the current thread can not be run yet. diff --git a/sys/sys/signalvar.h b/sys/sys/signalvar.h index 71685e7..68cca58 100644 --- a/sys/sys/signalvar.h +++ b/sys/sys/signalvar.h @@ -250,6 +250,20 @@ typedef struct sigqueue { /* Flags for ksi_flags */ #define SQ_INIT 0x01 =20 +/* + * Fast_sigblock + */ +#define FAST_SIGBLOCK_SETPTR 1 +#define FAST_SIGBLOCK_UNBLOCK 2 +#define FAST_SIGBLOCK_UNSETPTR 3 + +#define FAST_SIGBLOCK_PEND 0x1 +#define FAST_SIGBLOCK_INC 0x10 + +#ifndef _KERNEL +int __sys_fast_sigblock(int cmd, void *ptr); +#endif + #ifdef _KERNEL =20 /* Return nonzero if process p has an unmasked pending signal. */ @@ -329,6 +343,8 @@ extern struct mtx sigio_lock; =20 int cursig(struct thread *td, int stop_allowed); void execsigs(struct proc *p); +void fetch_fast_sigblock(struct thread *td); +void fetch_fast_sigblock_failed(struct thread *td, int write); void gsignal(int pgid, int sig, ksiginfo_t *ksi); void killproc(struct proc *p, char *why); ksiginfo_t * ksiginfo_alloc(int wait); ----- End forwarded message ----- --sdtB3X0nJg68CQEu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ6xJqAAoJEJDCuSvBvK1BiSMP/RrPt9uEu5jsbL+GbZ4EPALD veW06QvySP6TQ65jBFHoimZGXn3uSdduXEPBTqXsd5ECgbuptcy+luWORkc3b8Yz NuoTczRd6WJCC0J9WsxC3FgBVynq1iBsARfe4sg+SvpmlwnIkKON22E3FSy6k42z eYN/IDtYrC0nRzDSbBtkuIJ7QSXs2btY6vJHQhBMjhM/O7UtoZMF8tx/2T0khbvR Dt/HAM6HJBUYDOuYMvUJCx7NgogXcbHYDHD4FHpGriBmFNXxtAy/RkP0Xuv+9MO9 gQ5S22Uj/hkxwiz/nHMm+QNRNtqokywATLPm7LIwykMvPiwc0AwhXkKXFu87myjI Lk7aKFk/MuXERwZ6X8FU9ieodO4eJECgXF3Xsmkw3QxMk1KyhrslXbmyl7wLnuel dFHlOMAMjKx7Xka7sm3hZNIpztDJKS6cdys+VYYxHj2j0T45hEOO4Qk643dFb2sa a+DQdRi1p/dBY4fu1nmdRgGtbzl3u7ClqyEf9j+lWrLawgQ4QkWtUNlBBX8hohLz FuGNzMDRQs7UupUVHJfaHpoBKzRg1hhKczGux1pdd4wLTZBtvb47At4tVGquGyMT sDrN8vK/jPofhqTPg2IkRI1LnRQ6BZ5anwRiDmewO7QTyBkngO7Cdd58hirwQC2o 0PU6sx0kaeTntQH7hMVL =G0Up -----END PGP SIGNATURE----- --sdtB3X0nJg68CQEu-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 20:18:51 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 12DCA58C; Mon, 7 Jan 2013 20:18:51 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id AB44733C; Mon, 7 Jan 2013 20:18:50 +0000 (UTC) Received: from JRE-MBP-2.local (c-50-143-148-105.hsd1.ca.comcast.net [50.143.148.105]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id r07KImEt070010 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 7 Jan 2013 12:18:49 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <50EB2DA1.5090306@freebsd.org> Date: Mon, 07 Jan 2013 12:18:41 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> In-Reply-To: <20130107182235.GA65279@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 20:18:51 -0000 On 1/7/13 10:22 AM, Konstantin Belousov wrote: > Below is the forward of the patch for which I failed to obtain a private > review. Might be, the list generates more responses. > > Our rtld has a performance bootleneck, typically exposed by the images > with the lot of the run-time relocation processing, and by the C++ > exception handling. We block the signals delivery during the rtld > performing the relocations, as well as for the dl_iterate_phdr(3) (the > later is used for finding the dwarf unwinding tables). > > The signal blocking is needed to allow the rtld to resolve the symbols > for the signal handlers in the safe way, but also causes 2 syscalls > overhead per each rtld entry. > > The proposed approach allows to shave off those two syscalls, doubling > the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. > > Date: Mon, 13 Aug 2012 15:26:00 +0300 > From: Konstantin Belousov > > ... > > The basic idea is to implement sigprocmask() as single write into usermode > address. If kernel needs to calculate the signal mask for current thread, > it takes into the consideration non-zero value of the word at the agreed > address. Also, usermode is informed about signals which were put on hold > due to fast sigblock active. > > As I said, on my measurements in microbenchmark that did throw/catch in > a loop, I see equal user and system time spent for unpatched system, and > same user time with zero system time on patched system. > > Patch can be improved further, e.g. it would be nice to allow rtld to fall > back to sigprocmask(2) if kernel does not support fast sigblock, to prevent > flag day. Also, the mask enforced by fast sigblock can be made configurable. > > Note that libthr already blocks signals by catching them, and not using rtld > service in the first line handler. I tried to make the change in the spirit > of libthr interceptors, but handoff to libthr appears too complicated to > work. In fact, libthr can be changed to start using fast sigblock instead > of wrapping sigaction, but this is out of scope of the proposal right now. Is there any danger that an upriveliged user program can trick the kernel into doing anything it shouldn't by manipulating either the agreed upon address or the contents of the mask at the address? (even reading something by seeing what sigs get masked) This was an issue with the M:N threading package and resulted in extra syscalls to get around the problem. (I forget the details). From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 20:27:19 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DFBDE88D; Mon, 7 Jan 2013 20:27:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 3AA493DB; Mon, 7 Jan 2013 20:27:19 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r07KREqv078357; Mon, 7 Jan 2013 22:27:14 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua r07KREqv078357 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r07KREiw078356; Mon, 7 Jan 2013 22:27:14 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 7 Jan 2013 22:27:14 +0200 From: Konstantin Belousov To: Julian Elischer Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130107202714.GY82219@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EB2DA1.5090306@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6BJIGhnDLgQ8Wrmm" Content-Disposition: inline In-Reply-To: <50EB2DA1.5090306@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 20:27:19 -0000 --6BJIGhnDLgQ8Wrmm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 07, 2013 at 12:18:41PM -0800, Julian Elischer wrote: > On 1/7/13 10:22 AM, Konstantin Belousov wrote: > > Below is the forward of the patch for which I failed to obtain a private > > review. Might be, the list generates more responses. > > > > Our rtld has a performance bootleneck, typically exposed by the images > > with the lot of the run-time relocation processing, and by the C++ > > exception handling. We block the signals delivery during the rtld > > performing the relocations, as well as for the dl_iterate_phdr(3) (the > > later is used for finding the dwarf unwinding tables). > > > > The signal blocking is needed to allow the rtld to resolve the symbols > > for the signal handlers in the safe way, but also causes 2 syscalls > > overhead per each rtld entry. > > > > The proposed approach allows to shave off those two syscalls, doubling > > the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. > > > > Date: Mon, 13 Aug 2012 15:26:00 +0300 > > From: Konstantin Belousov > > > > ... > > > > The basic idea is to implement sigprocmask() as single write into userm= ode > > address. If kernel needs to calculate the signal mask for current threa= d, > > it takes into the consideration non-zero value of the word at the agreed > > address. Also, usermode is informed about signals which were put on hold > > due to fast sigblock active. > > > > As I said, on my measurements in microbenchmark that did throw/catch in > > a loop, I see equal user and system time spent for unpatched system, and > > same user time with zero system time on patched system. > > > > Patch can be improved further, e.g. it would be nice to allow rtld to f= all > > back to sigprocmask(2) if kernel does not support fast sigblock, to pre= vent > > flag day. Also, the mask enforced by fast sigblock can be made configur= able. > > > > Note that libthr already blocks signals by catching them, and not using= rtld > > service in the first line handler. I tried to make the change in the sp= irit > > of libthr interceptors, but handoff to libthr appears too complicated to > > work. In fact, libthr can be changed to start using fast sigblock inste= ad > > of wrapping sigaction, but this is out of scope of the proposal right n= ow. > Is there any danger that an upriveliged user program can trick the kernel > into doing anything it shouldn't by manipulating either the agreed upon > address or the contents of the mask at the address? (even reading > something by seeing what sigs get masked) I do not see how would anything like this possible. >=20 > This was an issue with the M:N threading package and resulted in extra=20 > syscalls > to get around the problem. (I forget the details). Would be great to dig up the details indeed. I cannot comment otherwise. --6BJIGhnDLgQ8Wrmm Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ6y+iAAoJEJDCuSvBvK1BaL4P/1vDmzlkJ6vxLZOwmTGbnoRe kaidamizUqMmlB2Fp4ruR2+6QeLn7oY2J0wSi7vtH/s+tynuQVSgfUgfxrqo68Lw 0cI4IWu/8TekYFLKlk3rDvpeHz8u3A3gzQOPXXwkutv07ZswJFN9moevMeAp+FA6 Kjh8VGzMaJACXnKk3qeSZEKWuyGYkcNA0xuY0TxtL4z6G6R856u2owmM1KA4iyC5 L9ZdaH2ZPCM8FWxJOBYbM7sIT4M99X9WB86RKGTBM7BgZdXG2yBU8f9KCTz4wkpP E9k985BJAcHLV8sf/1NQKG6ozMGRvCe1WAL2FoQ/7Cadr55ZbT8aPgh4xKcfKmWG v7FX3xhMZ11pufznNaJ05VChK/dnOM7EhXqIaV7nA99NTt5PebGbNVqCu16RxaUo hWuBDwN//JUmKGQz4Gzu/eVJLsOlKFGUX5AooskaG3+Dl6OL9sPcG9EfhAxLQRm6 VFOJ1jrfb+Ab3PEu44CYugl7OHIu3+rcXGNPI+mTLp3DmNEYes7u234fmO6aAqBl 8KWKf23WTdJjYRTrUSP6R/EKnfOi8yNVdCKXERn5ZPutYIQpTF+LuQRwyE4ct7do UMYTjXsVvu1DdY9LpMDp4fnXd4yKZaKXDs6pxkhvNriqvZvXB4Nmf/JCpxZEY1h5 JXDUqsOBRBj4FBKNV9A5 =JIfl -----END PGP SIGNATURE----- --6BJIGhnDLgQ8Wrmm-- From owner-freebsd-toolchain@FreeBSD.ORG Mon Jan 7 20:42:04 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2EE15D7F; Mon, 7 Jan 2013 20:42:04 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1AA496A1; Mon, 7 Jan 2013 20:42:03 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [207.238.187.242]) by elvis.mu.org (Postfix) with ESMTPSA id 4209E1A3EA7; Mon, 7 Jan 2013 12:42:02 -0800 (PST) Message-ID: <50EB3319.6060303@mu.org> Date: Mon, 07 Jan 2013 15:42:01 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> In-Reply-To: <20130107182235.GA65279@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 20:42:04 -0000 On 1/7/13 1:22 PM, Konstantin Belousov wrote: > Below is the forward of the patch for which I failed to obtain a private > review. Might be, the list generates more responses. Very cool. Sorry for being rusty here, but is it safe to call fuword in the middle of issignal()? The reason I ask is because it looks like proc lock is required for this block, and I forget it fuword(9) can fault. If fuword(9) can fault then I don't think you can do this. You'll need to use something like fuswintr(9) or wire the page down using vslock(9) or something. I am probably missing something? -Alfred From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 00:08:47 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 12DA7870; Tue, 8 Jan 2013 00:08:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 80799EAC; Tue, 8 Jan 2013 00:08:46 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r0808fMG010778; Tue, 8 Jan 2013 02:08:41 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0808fMG010778 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r0808e4J010777; Tue, 8 Jan 2013 02:08:40 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Jan 2013 02:08:40 +0200 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130108000840.GA82219@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EB3319.6060303@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/SXoEQDAiXmUSAYo" Content-Disposition: inline In-Reply-To: <50EB3319.6060303@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 00:08:47 -0000 --/SXoEQDAiXmUSAYo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 07, 2013 at 03:42:01PM -0500, Alfred Perlstein wrote: > On 1/7/13 1:22 PM, Konstantin Belousov wrote: > > Below is the forward of the patch for which I failed to obtain a private > > review. Might be, the list generates more responses. > Very cool. >=20 > Sorry for being rusty here, but is it safe to call fuword in the middle= =20 > of issignal()? >=20 > The reason I ask is because it looks like proc lock is required for this= =20 > block, and I forget it fuword(9) can fault. >=20 > If fuword(9) can fault then I don't think you can do this. >=20 > You'll need to use something like fuswintr(9) or wire the page down=20 > using vslock(9) or something. >=20 > I am probably missing something? No, you are completely right. This is a serious bug. Thank you. I was careful to code the td_sigblock_val prefetch etc, but failed to do the update properly. There is no reason to execute the update in the issignal() at all, the setting of the pending bit can be postponed to the moment after the postsig() loop was executed. I uploaded the updated but not yet tested patch at http://people.freebsd.org/~kib/misc/rtld-sigblock.1.patch --/SXoEQDAiXmUSAYo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ62OIAAoJEJDCuSvBvK1B3UwP/AtqY913PEsaOOuAuMCBfgo2 yYoBB/d4xaDzMNglQ6TxTNiqy4tN6Wd3vki38VrW4Kc4QbAmL4Yi+w08D8LkXqQ0 OuwP3PuHz6s9eoSOY+rIkHVhxG624SxnzCiKrSS6X8wT1Mky/dvYH/YsfgkXUTsH gbVJdP7P8jIkst9Rajepq6n9jjHqaWSJZ7JIMrQenVRy/x3PteidWGhL+UjIPBrw pOAXuwsvmr+NSqNhsiPCD7Q7yDOKBz3SALLjI/llc4cr1lC5j9C65bAa7gJKTHWZ Gj0jDz2+GI7CyM18TrUqMNspnh4WqXCSlAzhfsSG5LmS9Zl2FlI0T2UvOxLm9kl4 S5w4YofRVRUIjLyaXtebTGY29HE2mDripJkE6dYnU8sH0t7vShoLSfEvrHalBPNy px/Qtq8pRZWzPrnq/maFXCgqBwSfGEmuRhLX6e8EFHw35rnaN0f4p+cbqArJLpqB GIXksbN19Pre8LQfK4iNJxNqhUd0OniRqMP1qbqBU2/IrDTEmdhnT411fr1Wtvtz N1bLmEgou0Pgm5JC3h+FxbpnevvHaXgJGS01rt5L4GC/yAHmO60qf0KqbyRveMqu rxVaO7g1W+VeRpIxW9MUsk1dtyJosu/+higMc2iI063+YeZcvjRimLFOByHW7FXH BMfeFRMFZV7/ocl1IRdV =7eJr -----END PGP SIGNATURE----- --/SXoEQDAiXmUSAYo-- From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 05:09:24 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 99CBD333; Tue, 8 Jan 2013 05:09:24 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 746C8A7C; Tue, 8 Jan 2013 05:09:24 +0000 (UTC) Received: from xyf.my.dom (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r0859MdR082717; Tue, 8 Jan 2013 05:09:23 GMT (envelope-from davidxu@freebsd.org) Message-ID: <50EBAA1F.9070303@freebsd.org> Date: Tue, 08 Jan 2013 13:09:51 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:14.0) Gecko/20120822 Thunderbird/14.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> In-Reply-To: <20130107182235.GA65279@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 05:09:24 -0000 On 2013/01/08 02:22, Konstantin Belousov wrote: > Below is the forward of the patch for which I failed to obtain a private > review. Might be, the list generates more responses. > > Our rtld has a performance bootleneck, typically exposed by the images > with the lot of the run-time relocation processing, and by the C++ > exception handling. We block the signals delivery during the rtld > performing the relocations, as well as for the dl_iterate_phdr(3) (the > later is used for finding the dwarf unwinding tables). > > The signal blocking is needed to allow the rtld to resolve the symbols > for the signal handlers in the safe way, but also causes 2 syscalls > overhead per each rtld entry. > > The proposed approach allows to shave off those two syscalls, doubling > the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. > > Date: Mon, 13 Aug 2012 15:26:00 +0300 > From: Konstantin Belousov > > ... > > The basic idea is to implement sigprocmask() as single write into usermode > address. If kernel needs to calculate the signal mask for current thread, > it takes into the consideration non-zero value of the word at the agreed > address. Also, usermode is informed about signals which were put on hold > due to fast sigblock active. > > As I said, on my measurements in microbenchmark that did throw/catch in > a loop, I see equal user and system time spent for unpatched system, and > same user time with zero system time on patched system. > > Patch can be improved further, e.g. it would be nice to allow rtld to fall > back to sigprocmask(2) if kernel does not support fast sigblock, to prevent > flag day. Also, the mask enforced by fast sigblock can be made configurable. > > Note that libthr already blocks signals by catching them, and not using rtld > service in the first line handler. I tried to make the change in the spirit > of libthr interceptors, but handoff to libthr appears too complicated to > work. In fact, libthr can be changed to start using fast sigblock instead > of wrapping sigaction, but this is out of scope of the proposal right now. > > Please comment. > > diff --git a/lib/libc/sys/Symbol.map b/lib/libc/sys/Symbol.map > index 6888ea0..3b75539 100644 > --- a/lib/libc/sys/Symbol.map > +++ b/lib/libc/sys/Symbol.map > @@ -546,6 +546,7 @@ FBSDprivate_1.0 { > __sys_extattr_set_link; > _extattrctl; > __sys_extattrctl; > + __sys_fast_sigblock; > _fchdir; > __sys_fchdir; > _fchflags; > diff --git a/libexec/rtld-elf/rtld_lock.c b/libexec/rtld-elf/rtld_lock.c > index d1563e5..50c52c6 100644 > --- a/libexec/rtld-elf/rtld_lock.c > +++ b/libexec/rtld-elf/rtld_lock.c > @@ -43,6 +43,7 @@ > */ > > #include > +#include > #include > #include > #include > @@ -59,8 +60,8 @@ typedef struct Struct_Lock { > void *base; > } Lock; > > -static sigset_t fullsigmask, oldsigmask; > static int thread_flag; > +static uint32_t fsigblock; > > static void * > def_lock_create() > @@ -111,18 +112,26 @@ def_rlock_acquire(void *lock) > } > > static void > +sig_fastunblock(void) > +{ > + uint32_t oldval; > + > + oldval = atomic_fetchadd_32(&fsigblock, -FAST_SIGBLOCK_INC); > + if (oldval == (FAST_SIGBLOCK_PEND | FAST_SIGBLOCK_INC)) > + __sys_fast_sigblock(FAST_SIGBLOCK_UNBLOCK, NULL); > +} > + > +static void > def_wlock_acquire(void *lock) > { > Lock *l = (Lock *)lock; > - sigset_t tmp_oldsigmask; > > for ( ; ; ) { > - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); > + atomic_add_rel_32(&fsigblock, FAST_SIGBLOCK_INC); > if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) > break; > - sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); > + sig_fastunblock(); > } > - oldsigmask = tmp_oldsigmask; > } > > static void > @@ -134,7 +143,7 @@ def_lock_release(void *lock) > atomic_add_rel_int(&l->lock, -RC_INCR); > else { > atomic_add_rel_int(&l->lock, -WAFLAG); > - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); > + sig_fastunblock(); > } > } > > @@ -286,19 +295,7 @@ lockdflt_init() > > memcpy(&lockinfo, &deflockinfo, sizeof(lockinfo)); > _rtld_thread_init(NULL); > - /* > - * Construct a mask to block all signals except traps which might > - * conceivably be generated within the dynamic linker itself. > - */ > - sigfillset(&fullsigmask); > - sigdelset(&fullsigmask, SIGILL); > - sigdelset(&fullsigmask, SIGTRAP); > - sigdelset(&fullsigmask, SIGABRT); > - sigdelset(&fullsigmask, SIGEMT); > - sigdelset(&fullsigmask, SIGFPE); > - sigdelset(&fullsigmask, SIGBUS); > - sigdelset(&fullsigmask, SIGSEGV); > - sigdelset(&fullsigmask, SIGSYS); > + __sys_fast_sigblock(FAST_SIGBLOCK_SETPTR, &fsigblock); > } > > /* > @@ -319,7 +316,10 @@ _rtld_thread_init(struct RtldLockInfo *pli) > > if (pli == NULL) > pli = &deflockinfo; > - > + else { > + fsigblock = 0; > + __sys_fast_sigblock(FAST_SIGBLOCK_UNSETPTR, NULL); > + } > > for (i = 0; i < RTLD_LOCK_CNT; i++) > if ((locks[i] = pli->lock_create()) == NULL) > diff --git a/sys/compat/freebsd32/syscalls.master b/sys/compat/freebsd32/syscalls.master > index 0891e41..f9e8b9e 100644 > --- a/sys/compat/freebsd32/syscalls.master > +++ b/sys/compat/freebsd32/syscalls.master > @@ -997,3 +997,4 @@ > uint32_t offset1, uint32_t offset2,\ > uint32_t len1, uint32_t len2, \ > int advice); } > +532 AUE_NULL NOPROTO { int fast_sigblock(int cmd, uint32_t *ptr); } > diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c > index 90f7311..8a3cd15 100644 > --- a/sys/kern/kern_exec.c > +++ b/sys/kern/kern_exec.c > @@ -1031,6 +1031,7 @@ exec_new_vmspace(imgp, sv) > int error; > struct proc *p = imgp->proc; > struct vmspace *vmspace = p->p_vmspace; > + struct thread *td = curthread; > vm_object_t obj; > vm_offset_t sv_minuser, stack_addr; > vm_map_t map; > @@ -1039,6 +1040,10 @@ exec_new_vmspace(imgp, sv) > imgp->vmspace_destroyed = 1; > imgp->sysent = sv; > > + td->td_pflags &= ~TDP_FAST_SIGBLOCK; > + td->td_sigblock_ptr = NULL; > + td->td_sigblock_val = 0; > + > /* May be called with Giant held */ > EVENTHANDLER_INVOKE(process_exec, p, imgp); > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > index 2685a8b..3c8a2f7 100644 > --- a/sys/kern/kern_sig.c > +++ b/sys/kern/kern_sig.c > @@ -230,6 +230,7 @@ static int sigproptbl[NSIG] = { > }; > > static void reschedule_signals(struct proc *p, sigset_t block, int flags); > +static sigset_t fastblock_mask; > > static void > sigqueue_start(void) > @@ -240,6 +241,16 @@ sigqueue_start(void) > p31b_setcfg(CTL_P1003_1B_REALTIME_SIGNALS, _POSIX_REALTIME_SIGNALS); > p31b_setcfg(CTL_P1003_1B_RTSIG_MAX, SIGRTMAX - SIGRTMIN + 1); > p31b_setcfg(CTL_P1003_1B_SIGQUEUE_MAX, max_pending_per_proc); > + SIGFILLSET(fastblock_mask); > + SIG_CANTMASK(fastblock_mask); > + SIGDELSET(fastblock_mask, SIGILL); > + SIGDELSET(fastblock_mask, SIGTRAP); > + SIGDELSET(fastblock_mask, SIGABRT); > + SIGDELSET(fastblock_mask, SIGEMT); > + SIGDELSET(fastblock_mask, SIGFPE); > + SIGDELSET(fastblock_mask, SIGBUS); > + SIGDELSET(fastblock_mask, SIGSEGV); > + SIGDELSET(fastblock_mask, SIGSYS); > } > > ksiginfo_t * > @@ -2525,6 +2536,7 @@ issignal(struct thread *td, int stop_allowed) > struct sigqueue *queue; > sigset_t sigpending; > int sig, prop, newsig; > + uint32_t oldval; > > p = td->td_proc; > ps = p->p_sigacts; > @@ -2541,6 +2553,32 @@ issignal(struct thread *td, int stop_allowed) > SIG_STOPSIGMASK(sigpending); > if (SIGISEMPTY(sigpending)) /* no signal to send */ > return (0); > + > + /* > + * Do fast sigblock if requested by usermode. Since > + * we do know that there was a signal pending at this > + * point, set the FAST_SIGBLOCK_PEND as indicator for > + * usermode to perform a dummy call to > + * FAST_SIGBLOCK_UNBLOCK, which causes immediate > + * delivery of postponed pending signal. > + */ > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { > + if (td->td_sigblock_val != 0) > + SIGSETNAND(sigpending, fastblock_mask); > + if (SIGISEMPTY(sigpending)) { > + oldval = fuword32(td->td_sigblock_ptr); > + if (oldval == -1) { > + fetch_fast_sigblock_failed(td, 0); > + return (0); > + } > + oldval |= FAST_SIGBLOCK_PEND; > + if (suword32(td->td_sigblock_ptr, oldval) == -1) > + fetch_fast_sigblock_failed(td, 1); > + td->td_sigblock_val = oldval; > + return (0); > + } > + } > + > sig = sig_ffs(&sigpending); > > if (p->p_stops & S_SIG) { > @@ -3456,3 +3494,92 @@ sigacts_shared(struct sigacts *ps) > mtx_unlock(&ps->ps_mtx); > return (shared); > } > + > +int > +sys_fast_sigblock(struct thread *td, struct fast_sigblock_args *uap) > +{ > + struct proc *p; > + int error; > + uint32_t oldval; > + > + error = 0; > + switch (uap->cmd) { > + case FAST_SIGBLOCK_SETPTR: > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) > + error = EBUSY; > + else if (((uintptr_t)(uap->ptr) & (sizeof(uint32_t) - 1)) != 0) > + error = EINVAL; > + else { > + td->td_pflags |= TDP_FAST_SIGBLOCK; > + td->td_sigblock_ptr = uap->ptr; > + } > + break; > + > + case FAST_SIGBLOCK_UNBLOCK: > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { > + oldval = casuword32(td->td_sigblock_ptr, > + FAST_SIGBLOCK_PEND, 0); > + if (oldval == (uint32_t)-1) > + error = EFAULT; > + else if (oldval != FAST_SIGBLOCK_PEND) > + error = EBUSY; > + else > + td->td_sigblock_val = 0; > + } else > + error = EDOOFUS; > + > + /* > + * Rely on normal ast mechanism to deliver pending > + * signals to current thread. But notify others about > + * fake unblock. > + */ > + p = td->td_proc; > + if (error == 0 && p->p_numthreads != 1) { > + PROC_LOCK(p); > + reschedule_signals(p, td->td_sigmask, 0); > + PROC_UNLOCK(p); > + } > + break; > + > + case FAST_SIGBLOCK_UNSETPTR: > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { > + error = copyin(td->td_sigblock_ptr, &oldval, > + sizeof(uint32_t)); > + if (error != 0) > + break; > + if (oldval != 0 && oldval != FAST_SIGBLOCK_PEND) > + error = EBUSY; > + else { > + td->td_pflags &= ~TDP_FAST_SIGBLOCK; > + td->td_sigblock_val = 0; > + } > + } else > + error = EDOOFUS; > + break; > + } > + return (error); > +} > + > +void > +fetch_fast_sigblock(struct thread *td) > +{ > + > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) == 0) > + return; > + td->td_sigblock_val = fuword32(td->td_sigblock_ptr); > + if (td->td_sigblock_val == -1) > + fetch_fast_sigblock_failed(td, 0); > +} > + > +void > +fetch_fast_sigblock_failed(struct thread *td, int write) > +{ > + ksiginfo_t ksi; > + > + td->td_sigblock_val = 0; > + ksiginfo_init_trap(&ksi); > + ksi.ksi_signo = SIGSEGV; > + ksi.ksi_code = write ? SEGV_ACCERR : SEGV_MAPERR; > + ksi.ksi_addr = td->td_sigblock_ptr; > + trapsignal(td, &ksi); > +} > diff --git a/sys/kern/subr_syscall.c b/sys/kern/subr_syscall.c > index 5aee684..77b250d 100644 > --- a/sys/kern/subr_syscall.c > +++ b/sys/kern/subr_syscall.c > @@ -131,6 +131,12 @@ syscallenter(struct thread *td, struct syscall_args *sa) > sa->callp, sa->args, 0); > #endif > > + /* > + * Fetch fast sigblock value at the time of syscall > + * entry because sleepqueue primitives might call > + * cursig(). > + */ > + fetch_fast_sigblock(td); > AUDIT_SYSCALL_ENTER(sa->code, td); > error = (sa->callp->sy_call)(td, sa->args); > AUDIT_SYSCALL_EXIT(error, td); > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > index 095bbdf..66e485b 100644 > --- a/sys/kern/subr_trap.c > +++ b/sys/kern/subr_trap.c > @@ -237,6 +237,7 @@ ast(struct trapframe *framep) > */ > if (flags & TDF_NEEDSIGCHK || p->p_pendingcnt > 0 || > !SIGISEMPTY(p->p_siglist)) { > + fetch_fast_sigblock(td); > PROC_LOCK(p); > mtx_lock(&p->p_sigacts->ps_mtx); > while ((sig = cursig(td, SIG_STOP_ALLOWED)) != 0) > diff --git a/sys/kern/syscalls.master b/sys/kern/syscalls.master > index f62dad7..28a9393 100644 > --- a/sys/kern/syscalls.master > +++ b/sys/kern/syscalls.master > @@ -951,5 +951,6 @@ > off_t offset, off_t len); } > 531 AUE_NULL STD { int posix_fadvise(int fd, off_t offset, \ > off_t len, int advice); } > +532 AUE_NULL STD { int fast_sigblock(int cmd, uint32_t *ptr); } > ; Please copy any additions and changes to the following compatability tables: > ; sys/compat/freebsd32/syscalls.master > diff --git a/sys/sys/proc.h b/sys/sys/proc.h > index 06df632..4899ca2 100644 > --- a/sys/sys/proc.h > +++ b/sys/sys/proc.h > @@ -272,6 +272,9 @@ struct thread { > struct osd td_osd; /* (k) Object specific data. */ > struct vm_map_entry *td_map_def_user; /* (k) Deferred entries. */ > pid_t td_dbg_forked; /* (c) Child pid for debugger. */ > + void *td_sigblock_ptr; /* (k) uptr for fast sigblock. */ > + uint32_t td_sigblock_val; /* (k) fast sigblock value at > + kernel entry. */ > #define td_endzero td_sigmask > > /* Copied during fork1() or create_thread(). */ > @@ -424,6 +427,7 @@ do { \ > #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */ > #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ > #define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */ > +#define TDP_FAST_SIGBLOCK 0x20000000 /* Fast sigblock active */ > > /* > * Reasons that the current thread can not be run yet. > diff --git a/sys/sys/signalvar.h b/sys/sys/signalvar.h > index 71685e7..68cca58 100644 > --- a/sys/sys/signalvar.h > +++ b/sys/sys/signalvar.h > @@ -250,6 +250,20 @@ typedef struct sigqueue { > /* Flags for ksi_flags */ > #define SQ_INIT 0x01 > > +/* > + * Fast_sigblock > + */ > +#define FAST_SIGBLOCK_SETPTR 1 > +#define FAST_SIGBLOCK_UNBLOCK 2 > +#define FAST_SIGBLOCK_UNSETPTR 3 > + > +#define FAST_SIGBLOCK_PEND 0x1 > +#define FAST_SIGBLOCK_INC 0x10 > + > +#ifndef _KERNEL > +int __sys_fast_sigblock(int cmd, void *ptr); > +#endif > + > #ifdef _KERNEL > > /* Return nonzero if process p has an unmasked pending signal. */ > @@ -329,6 +343,8 @@ extern struct mtx sigio_lock; > > int cursig(struct thread *td, int stop_allowed); > void execsigs(struct proc *p); > +void fetch_fast_sigblock(struct thread *td); > +void fetch_fast_sigblock_failed(struct thread *td, int write); > void gsignal(int pgid, int sig, ksiginfo_t *ksi); > void killproc(struct proc *p, char *why); > ksiginfo_t * ksiginfo_alloc(int wait); > > > > ----- End forwarded message ----- > So you want to make kernel share data with userland. The only thing I don't like is it adds overhead to syscall, rumor said that our syscall is slow than others, this might not be a problem ? Long time ago, I proposed a schedctl() syscall to make kernel share data with userland, some old patches are still there: http://people.freebsd.org/~davidxu/schedctl/ Mine does not have such an overhead, but it has another one: memory page is allocated in kernel which can not be swapped out and can not be freed until process is exited, the page is doubly mapped into in kernel and userland, accessing the shared data in kernel has zero overhead though. From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 06:21:54 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB7C570D; Tue, 8 Jan 2013 06:21:54 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id CAE67D30; Tue, 8 Jan 2013 06:21:54 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [207.238.187.242]) by elvis.mu.org (Postfix) with ESMTPSA id 501121A3CC4; Mon, 7 Jan 2013 22:21:51 -0800 (PST) Message-ID: <50EBBAFC.9070803@mu.org> Date: Tue, 08 Jan 2013 01:21:48 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <50EB3319.6060303@mu.org> <20130108000840.GA82219@kib.kiev.ua> In-Reply-To: <20130108000840.GA82219@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 06:21:54 -0000 On 1/7/13 7:08 PM, Konstantin Belousov wrote: > On Mon, Jan 07, 2013 at 03:42:01PM -0500, Alfred Perlstein wrote: >> On 1/7/13 1:22 PM, Konstantin Belousov wrote: >>> Below is the forward of the patch for which I failed to obtain a private >>> review. Might be, the list generates more responses. >> Very cool. >> >> Sorry for being rusty here, but is it safe to call fuword in the middle >> of issignal()? >> >> The reason I ask is because it looks like proc lock is required for this >> block, and I forget it fuword(9) can fault. >> >> If fuword(9) can fault then I don't think you can do this. >> >> You'll need to use something like fuswintr(9) or wire the page down >> using vslock(9) or something. >> >> I am probably missing something? > No, you are completely right. This is a serious bug. Thank you. > > I was careful to code the td_sigblock_val prefetch etc, but failed > to do the update properly. There is no reason to execute the update > in the issignal() at all, the setting of the pending bit can be postponed > to the moment after the postsig() loop was executed. > > I uploaded the updated but not yet tested patch at > http://people.freebsd.org/~kib/misc/rtld-sigblock.1.patch This new patch looks like it may have issues with PSTOP. there is a lot of code in issignal() that is missed when this code is in place, I have not audited the effect of this, are you sure this is what will work for the majority of cases? Honestly, if I were coding it for correctness I would vslock the pages (or otherwise wire them in) and let issignal() behave normally and not early exit from it. That said I may be missing something. -Alfred From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 14:56:39 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CC1A078; Tue, 8 Jan 2013 14:56:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id EE3CB729; Tue, 8 Jan 2013 14:56:38 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r08EuVV9096773; Tue, 8 Jan 2013 16:56:31 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r08EuVV9096773 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r08EuVef096772; Tue, 8 Jan 2013 16:56:31 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Jan 2013 16:56:31 +0200 From: Konstantin Belousov To: David Xu Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130108145631.GD82219@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="em00FKA6+IJh3lGW" Content-Disposition: inline In-Reply-To: <50EBAA1F.9070303@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 14:56:39 -0000 --em00FKA6+IJh3lGW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 08, 2013 at 01:09:51PM +0800, David Xu wrote: > On 2013/01/08 02:22, Konstantin Belousov wrote: > > Below is the forward of the patch for which I failed to obtain a private > > review. Might be, the list generates more responses. > > > > Our rtld has a performance bootleneck, typically exposed by the images > > with the lot of the run-time relocation processing, and by the C++ > > exception handling. We block the signals delivery during the rtld > > performing the relocations, as well as for the dl_iterate_phdr(3) (the > > later is used for finding the dwarf unwinding tables). > > > > The signal blocking is needed to allow the rtld to resolve the symbols > > for the signal handlers in the safe way, but also causes 2 syscalls > > overhead per each rtld entry. > > > > The proposed approach allows to shave off those two syscalls, doubling > > the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. > > > > Date: Mon, 13 Aug 2012 15:26:00 +0300 > > From: Konstantin Belousov > > > > ... > > > > The basic idea is to implement sigprocmask() as single write into userm= ode > > address. If kernel needs to calculate the signal mask for current threa= d, > > it takes into the consideration non-zero value of the word at the agreed > > address. Also, usermode is informed about signals which were put on hold > > due to fast sigblock active. > > > > As I said, on my measurements in microbenchmark that did throw/catch in > > a loop, I see equal user and system time spent for unpatched system, and > > same user time with zero system time on patched system. > > > > Patch can be improved further, e.g. it would be nice to allow rtld to f= all > > back to sigprocmask(2) if kernel does not support fast sigblock, to pre= vent > > flag day. Also, the mask enforced by fast sigblock can be made configur= able. > > > > Note that libthr already blocks signals by catching them, and not using= rtld > > service in the first line handler. I tried to make the change in the sp= irit > > of libthr interceptors, but handoff to libthr appears too complicated to > > work. In fact, libthr can be changed to start using fast sigblock inste= ad > > of wrapping sigaction, but this is out of scope of the proposal right n= ow. > > > > Please comment. > > > > diff --git a/lib/libc/sys/Symbol.map b/lib/libc/sys/Symbol.map > > index 6888ea0..3b75539 100644 > > --- a/lib/libc/sys/Symbol.map > > +++ b/lib/libc/sys/Symbol.map > > @@ -546,6 +546,7 @@ FBSDprivate_1.0 { > > __sys_extattr_set_link; > > _extattrctl; > > __sys_extattrctl; > > + __sys_fast_sigblock; > > _fchdir; > > __sys_fchdir; > > _fchflags; > > diff --git a/libexec/rtld-elf/rtld_lock.c b/libexec/rtld-elf/rtld_lock.c > > index d1563e5..50c52c6 100644 > > --- a/libexec/rtld-elf/rtld_lock.c > > +++ b/libexec/rtld-elf/rtld_lock.c > > @@ -43,6 +43,7 @@ > > */ > > > > #include > > +#include > > #include > > #include > > #include > > @@ -59,8 +60,8 @@ typedef struct Struct_Lock { > > void *base; > > } Lock; > > > > -static sigset_t fullsigmask, oldsigmask; > > static int thread_flag; > > +static uint32_t fsigblock; > > > > static void * > > def_lock_create() > > @@ -111,18 +112,26 @@ def_rlock_acquire(void *lock) > > } > > > > static void > > +sig_fastunblock(void) > > +{ > > + uint32_t oldval; > > + > > + oldval =3D atomic_fetchadd_32(&fsigblock, -FAST_SIGBLOCK_INC); > > + if (oldval =3D=3D (FAST_SIGBLOCK_PEND | FAST_SIGBLOCK_INC)) > > + __sys_fast_sigblock(FAST_SIGBLOCK_UNBLOCK, NULL); > > +} > > + > > +static void > > def_wlock_acquire(void *lock) > > { > > Lock *l =3D (Lock *)lock; > > - sigset_t tmp_oldsigmask; > > > > for ( ; ; ) { > > - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); > > + atomic_add_rel_32(&fsigblock, FAST_SIGBLOCK_INC); > > if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) > > break; > > - sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); > > + sig_fastunblock(); > > } > > - oldsigmask =3D tmp_oldsigmask; > > } > > > > static void > > @@ -134,7 +143,7 @@ def_lock_release(void *lock) > > atomic_add_rel_int(&l->lock, -RC_INCR); > > else { > > atomic_add_rel_int(&l->lock, -WAFLAG); > > - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); > > + sig_fastunblock(); > > } > > } > > > > @@ -286,19 +295,7 @@ lockdflt_init() > > > > memcpy(&lockinfo, &deflockinfo, sizeof(lockinfo)); > > _rtld_thread_init(NULL); > > - /* > > - * Construct a mask to block all signals except traps which might > > - * conceivably be generated within the dynamic linker itself. > > - */ > > - sigfillset(&fullsigmask); > > - sigdelset(&fullsigmask, SIGILL); > > - sigdelset(&fullsigmask, SIGTRAP); > > - sigdelset(&fullsigmask, SIGABRT); > > - sigdelset(&fullsigmask, SIGEMT); > > - sigdelset(&fullsigmask, SIGFPE); > > - sigdelset(&fullsigmask, SIGBUS); > > - sigdelset(&fullsigmask, SIGSEGV); > > - sigdelset(&fullsigmask, SIGSYS); > > + __sys_fast_sigblock(FAST_SIGBLOCK_SETPTR, &fsigblock); > > } > > > > /* > > @@ -319,7 +316,10 @@ _rtld_thread_init(struct RtldLockInfo *pli) > > > > if (pli =3D=3D NULL) > > pli =3D &deflockinfo; > > - > > + else { > > + fsigblock =3D 0; > > + __sys_fast_sigblock(FAST_SIGBLOCK_UNSETPTR, NULL); > > + } > > > > for (i =3D 0; i < RTLD_LOCK_CNT; i++) > > if ((locks[i] =3D pli->lock_create()) =3D=3D NULL) > > diff --git a/sys/compat/freebsd32/syscalls.master b/sys/compat/freebsd3= 2/syscalls.master > > index 0891e41..f9e8b9e 100644 > > --- a/sys/compat/freebsd32/syscalls.master > > +++ b/sys/compat/freebsd32/syscalls.master > > @@ -997,3 +997,4 @@ > > uint32_t offset1, uint32_t offset2,\ > > uint32_t len1, uint32_t len2, \ > > int advice); } > > +532 AUE_NULL NOPROTO { int fast_sigblock(int cmd, uint32_t *ptr); } > > diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c > > index 90f7311..8a3cd15 100644 > > --- a/sys/kern/kern_exec.c > > +++ b/sys/kern/kern_exec.c > > @@ -1031,6 +1031,7 @@ exec_new_vmspace(imgp, sv) > > int error; > > struct proc *p =3D imgp->proc; > > struct vmspace *vmspace =3D p->p_vmspace; > > + struct thread *td =3D curthread; > > vm_object_t obj; > > vm_offset_t sv_minuser, stack_addr; > > vm_map_t map; > > @@ -1039,6 +1040,10 @@ exec_new_vmspace(imgp, sv) > > imgp->vmspace_destroyed =3D 1; > > imgp->sysent =3D sv; > > > > + td->td_pflags &=3D ~TDP_FAST_SIGBLOCK; > > + td->td_sigblock_ptr =3D NULL; > > + td->td_sigblock_val =3D 0; > > + > > /* May be called with Giant held */ > > EVENTHANDLER_INVOKE(process_exec, p, imgp); > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > index 2685a8b..3c8a2f7 100644 > > --- a/sys/kern/kern_sig.c > > +++ b/sys/kern/kern_sig.c > > @@ -230,6 +230,7 @@ static int sigproptbl[NSIG] =3D { > > }; > > > > static void reschedule_signals(struct proc *p, sigset_t block, int fl= ags); > > +static sigset_t fastblock_mask; > > > > static void > > sigqueue_start(void) > > @@ -240,6 +241,16 @@ sigqueue_start(void) > > p31b_setcfg(CTL_P1003_1B_REALTIME_SIGNALS, _POSIX_REALTIME_SIGNALS); > > p31b_setcfg(CTL_P1003_1B_RTSIG_MAX, SIGRTMAX - SIGRTMIN + 1); > > p31b_setcfg(CTL_P1003_1B_SIGQUEUE_MAX, max_pending_per_proc); > > + SIGFILLSET(fastblock_mask); > > + SIG_CANTMASK(fastblock_mask); > > + SIGDELSET(fastblock_mask, SIGILL); > > + SIGDELSET(fastblock_mask, SIGTRAP); > > + SIGDELSET(fastblock_mask, SIGABRT); > > + SIGDELSET(fastblock_mask, SIGEMT); > > + SIGDELSET(fastblock_mask, SIGFPE); > > + SIGDELSET(fastblock_mask, SIGBUS); > > + SIGDELSET(fastblock_mask, SIGSEGV); > > + SIGDELSET(fastblock_mask, SIGSYS); > > } > > > > ksiginfo_t * > > @@ -2525,6 +2536,7 @@ issignal(struct thread *td, int stop_allowed) > > struct sigqueue *queue; > > sigset_t sigpending; > > int sig, prop, newsig; > > + uint32_t oldval; > > > > p =3D td->td_proc; > > ps =3D p->p_sigacts; > > @@ -2541,6 +2553,32 @@ issignal(struct thread *td, int stop_allowed) > > SIG_STOPSIGMASK(sigpending); > > if (SIGISEMPTY(sigpending)) /* no signal to send */ > > return (0); > > + > > + /* > > + * Do fast sigblock if requested by usermode. Since > > + * we do know that there was a signal pending at this > > + * point, set the FAST_SIGBLOCK_PEND as indicator for > > + * usermode to perform a dummy call to > > + * FAST_SIGBLOCK_UNBLOCK, which causes immediate > > + * delivery of postponed pending signal. > > + */ > > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { > > + if (td->td_sigblock_val !=3D 0) > > + SIGSETNAND(sigpending, fastblock_mask); > > + if (SIGISEMPTY(sigpending)) { > > + oldval =3D fuword32(td->td_sigblock_ptr); > > + if (oldval =3D=3D -1) { > > + fetch_fast_sigblock_failed(td, 0); > > + return (0); > > + } > > + oldval |=3D FAST_SIGBLOCK_PEND; > > + if (suword32(td->td_sigblock_ptr, oldval) =3D=3D -1) > > + fetch_fast_sigblock_failed(td, 1); > > + td->td_sigblock_val =3D oldval; > > + return (0); > > + } > > + } > > + > > sig =3D sig_ffs(&sigpending); > > > > if (p->p_stops & S_SIG) { > > @@ -3456,3 +3494,92 @@ sigacts_shared(struct sigacts *ps) > > mtx_unlock(&ps->ps_mtx); > > return (shared); > > } > > + > > +int > > +sys_fast_sigblock(struct thread *td, struct fast_sigblock_args *uap) > > +{ > > + struct proc *p; > > + int error; > > + uint32_t oldval; > > + > > + error =3D 0; > > + switch (uap->cmd) { > > + case FAST_SIGBLOCK_SETPTR: > > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) > > + error =3D EBUSY; > > + else if (((uintptr_t)(uap->ptr) & (sizeof(uint32_t) - 1)) !=3D 0) > > + error =3D EINVAL; > > + else { > > + td->td_pflags |=3D TDP_FAST_SIGBLOCK; > > + td->td_sigblock_ptr =3D uap->ptr; > > + } > > + break; > > + > > + case FAST_SIGBLOCK_UNBLOCK: > > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { > > + oldval =3D casuword32(td->td_sigblock_ptr, > > + FAST_SIGBLOCK_PEND, 0); > > + if (oldval =3D=3D (uint32_t)-1) > > + error =3D EFAULT; > > + else if (oldval !=3D FAST_SIGBLOCK_PEND) > > + error =3D EBUSY; > > + else > > + td->td_sigblock_val =3D 0; > > + } else > > + error =3D EDOOFUS; > > + > > + /* > > + * Rely on normal ast mechanism to deliver pending > > + * signals to current thread. But notify others about > > + * fake unblock. > > + */ > > + p =3D td->td_proc; > > + if (error =3D=3D 0 && p->p_numthreads !=3D 1) { > > + PROC_LOCK(p); > > + reschedule_signals(p, td->td_sigmask, 0); > > + PROC_UNLOCK(p); > > + } > > + break; > > + > > + case FAST_SIGBLOCK_UNSETPTR: > > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) !=3D 0) { > > + error =3D copyin(td->td_sigblock_ptr, &oldval, > > + sizeof(uint32_t)); > > + if (error !=3D 0) > > + break; > > + if (oldval !=3D 0 && oldval !=3D FAST_SIGBLOCK_PEND) > > + error =3D EBUSY; > > + else { > > + td->td_pflags &=3D ~TDP_FAST_SIGBLOCK; > > + td->td_sigblock_val =3D 0; > > + } > > + } else > > + error =3D EDOOFUS; > > + break; > > + } > > + return (error); > > +} > > + > > +void > > +fetch_fast_sigblock(struct thread *td) > > +{ > > + > > + if ((td->td_pflags & TDP_FAST_SIGBLOCK) =3D=3D 0) > > + return; > > + td->td_sigblock_val =3D fuword32(td->td_sigblock_ptr); > > + if (td->td_sigblock_val =3D=3D -1) > > + fetch_fast_sigblock_failed(td, 0); > > +} > > + > > +void > > +fetch_fast_sigblock_failed(struct thread *td, int write) > > +{ > > + ksiginfo_t ksi; > > + > > + td->td_sigblock_val =3D 0; > > + ksiginfo_init_trap(&ksi); > > + ksi.ksi_signo =3D SIGSEGV; > > + ksi.ksi_code =3D write ? SEGV_ACCERR : SEGV_MAPERR; > > + ksi.ksi_addr =3D td->td_sigblock_ptr; > > + trapsignal(td, &ksi); > > +} > > diff --git a/sys/kern/subr_syscall.c b/sys/kern/subr_syscall.c > > index 5aee684..77b250d 100644 > > --- a/sys/kern/subr_syscall.c > > +++ b/sys/kern/subr_syscall.c > > @@ -131,6 +131,12 @@ syscallenter(struct thread *td, struct syscall_arg= s *sa) > > sa->callp, sa->args, 0); > > #endif > > > > + /* > > + * Fetch fast sigblock value at the time of syscall > > + * entry because sleepqueue primitives might call > > + * cursig(). > > + */ > > + fetch_fast_sigblock(td); > > AUDIT_SYSCALL_ENTER(sa->code, td); > > error =3D (sa->callp->sy_call)(td, sa->args); > > AUDIT_SYSCALL_EXIT(error, td); > > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > > index 095bbdf..66e485b 100644 > > --- a/sys/kern/subr_trap.c > > +++ b/sys/kern/subr_trap.c > > @@ -237,6 +237,7 @@ ast(struct trapframe *framep) > > */ > > if (flags & TDF_NEEDSIGCHK || p->p_pendingcnt > 0 || > > !SIGISEMPTY(p->p_siglist)) { > > + fetch_fast_sigblock(td); > > PROC_LOCK(p); > > mtx_lock(&p->p_sigacts->ps_mtx); > > while ((sig =3D cursig(td, SIG_STOP_ALLOWED)) !=3D 0) > > diff --git a/sys/kern/syscalls.master b/sys/kern/syscalls.master > > index f62dad7..28a9393 100644 > > --- a/sys/kern/syscalls.master > > +++ b/sys/kern/syscalls.master > > @@ -951,5 +951,6 @@ > > off_t offset, off_t len); } > > 531 AUE_NULL STD { int posix_fadvise(int fd, off_t offset, \ > > off_t len, int advice); } > > +532 AUE_NULL STD { int fast_sigblock(int cmd, uint32_t *ptr); } > > ; Please copy any additions and changes to the following compatabilit= y tables: > > ; sys/compat/freebsd32/syscalls.master > > diff --git a/sys/sys/proc.h b/sys/sys/proc.h > > index 06df632..4899ca2 100644 > > --- a/sys/sys/proc.h > > +++ b/sys/sys/proc.h > > @@ -272,6 +272,9 @@ struct thread { > > struct osd td_osd; /* (k) Object specific data. */ > > struct vm_map_entry *td_map_def_user; /* (k) Deferred entries. */ > > pid_t td_dbg_forked; /* (c) Child pid for debugger. */ > > + void *td_sigblock_ptr; /* (k) uptr for fast sigblock. */ > > + uint32_t td_sigblock_val; /* (k) fast sigblock value at > > + kernel entry. */ > > #define td_endzero td_sigmask > > > > /* Copied during fork1() or create_thread(). */ > > @@ -424,6 +427,7 @@ do { \ > > #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history= =2E */ > > #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ > > #define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma= */ > > +#define TDP_FAST_SIGBLOCK 0x20000000 /* Fast sigblock active */ > > > > /* > > * Reasons that the current thread can not be run yet. > > diff --git a/sys/sys/signalvar.h b/sys/sys/signalvar.h > > index 71685e7..68cca58 100644 > > --- a/sys/sys/signalvar.h > > +++ b/sys/sys/signalvar.h > > @@ -250,6 +250,20 @@ typedef struct sigqueue { > > /* Flags for ksi_flags */ > > #define SQ_INIT 0x01 > > > > +/* > > + * Fast_sigblock > > + */ > > +#define FAST_SIGBLOCK_SETPTR 1 > > +#define FAST_SIGBLOCK_UNBLOCK 2 > > +#define FAST_SIGBLOCK_UNSETPTR 3 > > + > > +#define FAST_SIGBLOCK_PEND 0x1 > > +#define FAST_SIGBLOCK_INC 0x10 > > + > > +#ifndef _KERNEL > > +int __sys_fast_sigblock(int cmd, void *ptr); > > +#endif > > + > > #ifdef _KERNEL > > > > /* Return nonzero if process p has an unmasked pending signal. */ > > @@ -329,6 +343,8 @@ extern struct mtx sigio_lock; > > > > int cursig(struct thread *td, int stop_allowed); > > void execsigs(struct proc *p); > > +void fetch_fast_sigblock(struct thread *td); > > +void fetch_fast_sigblock_failed(struct thread *td, int write); > > void gsignal(int pgid, int sig, ksiginfo_t *ksi); > > void killproc(struct proc *p, char *why); > > ksiginfo_t * ksiginfo_alloc(int wait); > > > > > > > > ----- End forwarded message ----- > > > So you want to make kernel share data with userland. The only thing I > don't like is it adds overhead to syscall, rumor said that our syscall > is slow than others, this might not be a problem ? >=20 > Long time ago, I proposed a schedctl() syscall to make kernel share > data with userland, some old patches are still there: > http://people.freebsd.org/~davidxu/schedctl/ >=20 > Mine does not have such an overhead, but it has another one: > memory page is allocated in kernel which can not be swapped out > and can not be freed until process is exited, the page is doubly > mapped into in kernel and userland, accessing the shared data > in kernel has zero overhead though. Initial version of the changes indeed used the remap of the user page into the KVA. Besides the issues you described regarding the page being wired for the whole life of the thread, as well as the one page frame in the KVA, there were other non-trivial problems. It was mostly related to the fork(2) copying the address spaces, but not limited too. The use of the direct user address access appeared to be much less intrusive. I completely agree with your note about the additional memory access in the syscall entry path. I do believe that it does not incur a overhead in the real-world loads, but might make some skews in the microbenchmarks. I did not noted any in the testing I did. Anyway, I am still not sure is it worth optimizing for the throw/catch microbenchmark at ll. If general public consequence is yes, the patch needs some work before being committable anyway, mostly to allow patched rtld to work on the pre-patch kernels. Thank you for the reply. --em00FKA6+IJh3lGW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ7DOeAAoJEJDCuSvBvK1B2gAP/0J5YxA+X//ck3myMOOFSGWE Tai1FZe99/kWOiq5Hz6ytPmY1N2enNIePAwboPEPXcqRRu15xG5Pa4nmJqv68a98 Ue3cd6Jf2M/wARRwjWBUjbd/unQWFzQqnXRxVRZJ0F7wFwFS+49hpxLx90UHK3eM g0u4/WCKhf2Czg4rzLSo4L+IhIJ5y+Pw3tkyEfKFe3phJaFdZEhkhZ36WpDqwC6P IQI3kYbKdKIs74zOVpyFn4LBvCk67lGAVF/UF2XVsDreiGzE4w6TfUShmcXAlc+5 nU7la9yvVItmwu7QoONVdi5ccRmO/yZVClGGxPCI/yH/JCdbzNnnqHXlUjRJnI2K U7uUEniyb8NblmK7j1OhouoyjukcspVL/oFhsh+1SAaMVSfY1IXNVvGkN8OqG5ID fgVMtwesYQM+ajGK7CMubA+1e7R2RYbfMrCZ2RuCSDEUqKc2zBH2blr3LfN0Hfnd REtIqtQ2GgA0TdZghT1f+MyNrKpYSyAAxQc/C4FbLdtUzPTotAMDOOE+HIYDxGs9 CnCPmJi7CuDLCjjOGFmQmt3iXMZ3iB96W1mt71931n+V3x6kuY56sU3mYQ0rWIJn jFFkmJyFnfaSmTgAcPCjNuQB5fL4HmgIwSnkTNofYnMvrHPVZJKrF3wAc74lNlG9 YSmACN9ERrLG1Blenu0D =gbJr -----END PGP SIGNATURE----- --em00FKA6+IJh3lGW-- From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 15:00:35 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 112AC1CA; Tue, 8 Jan 2013 15:00:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 7ED3D74F; Tue, 8 Jan 2013 15:00:34 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r08F0PSP097265; Tue, 8 Jan 2013 17:00:25 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r08F0PSP097265 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r08F0OwQ097264; Tue, 8 Jan 2013 17:00:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Jan 2013 17:00:24 +0200 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130108150024.GE82219@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EB3319.6060303@mu.org> <20130108000840.GA82219@kib.kiev.ua> <50EBBAFC.9070803@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="prKJgtdAmn5IQEeG" Content-Disposition: inline In-Reply-To: <50EBBAFC.9070803@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 15:00:35 -0000 --prKJgtdAmn5IQEeG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 08, 2013 at 01:21:48AM -0500, Alfred Perlstein wrote: > On 1/7/13 7:08 PM, Konstantin Belousov wrote: > > On Mon, Jan 07, 2013 at 03:42:01PM -0500, Alfred Perlstein wrote: > >> On 1/7/13 1:22 PM, Konstantin Belousov wrote: > >>> Below is the forward of the patch for which I failed to obtain a priv= ate > >>> review. Might be, the list generates more responses. > >> Very cool. > >> > >> Sorry for being rusty here, but is it safe to call fuword in the middle > >> of issignal()? > >> > >> The reason I ask is because it looks like proc lock is required for th= is > >> block, and I forget it fuword(9) can fault. > >> > >> If fuword(9) can fault then I don't think you can do this. > >> > >> You'll need to use something like fuswintr(9) or wire the page down > >> using vslock(9) or something. > >> > >> I am probably missing something? > > No, you are completely right. This is a serious bug. Thank you. > > > > I was careful to code the td_sigblock_val prefetch etc, but failed > > to do the update properly. There is no reason to execute the update > > in the issignal() at all, the setting of the pending bit can be postpon= ed > > to the moment after the postsig() loop was executed. > > > > I uploaded the updated but not yet tested patch at > > http://people.freebsd.org/~kib/misc/rtld-sigblock.1.patch >=20 > This new patch looks like it may have issues with PSTOP. >=20 > there is a lot of code in issignal() that is missed when this code is in= =20 > place, I have not audited the effect of this, are you sure this is what= =20 > will work for the majority of cases? What exactly you are concerned with ? Note that fastblock_mask manipulations right before setting of TDP_FAST_SIGPENDING is to disallow the postponing of the delivery of (potentially) synchronous signals caused by traps, as well as SIGKILL/SIGSTOP. >=20 > Honestly, if I were coding it for correctness I would vslock the pages=20 > (or otherwise wire them in) and let issignal() behave normally and not=20 > early exit from it. vslock() is racy, causes user address space fragmentation and was simply impossibly to use correctly until very recent. --prKJgtdAmn5IQEeG Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ7DSHAAoJEJDCuSvBvK1BRvwP/01DeP1jOWHj4dsGCavZPXKh D3yI6BYAaPxiQihncwRQg8FO/eSoXlAx16wRbrZz5aj7FUq6taOEJ5mcf6jkp1mr IM47VZVQnYWunDP7rrLP4EsurIPfn5OVyazx/hXQO4lD+6JFpWczGP5FQUjJDWK4 2TdMvYa/Q3Wfj73xzVuvtKncrwEx3klT0LOq+fPvUBP/I2AWHI/Wh2+yZVcHD3PL 9mihrB5JuK+h6chbKYmNGrNSoAZQqNBXDHWoGtSfIlPr1jfzzl+Pdq9Z0cOJWYyY qZOs5GksPHbwNWQc2qJrKgLA/ckZaZF8CLAlxWpspizsBoqztQGE9lVPQGP9rVOz J2PbUzPZsfUNzjHvsT1mIoicC/rl1Q98IgKc521npXgCKza9nzQXOxi4rCNZ4Uoo wueGh/y+OP75cTSn88/QZS46pM3gXi+G2Udx46hRJO0rrCYKocGlEG2nHRtULE8S 45VwUciiswuVVwWoKw9w93MCIzCtvX0Hv0GaRoJrz8lNJ9Dzo/uqGrxXkUdv8BbT Z2rFwRoeddXi8j8cmKmAg/bSfooKJAocndPSleaakmSG54oijAxSt3KIqf4QJrw4 ZuTmY240QUZaBbQkkqUltlEL06dF4G+vZ1pvKcZ5KCtr4T9ICp5VCVacQMcsYQ9Q 7/t9PLEpX6cmyozwydb9 =5W7k -----END PGP SIGNATURE----- --prKJgtdAmn5IQEeG-- From owner-freebsd-toolchain@FreeBSD.ORG Tue Jan 8 15:02:30 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5170239B; Tue, 8 Jan 2013 15:02:30 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 2C12677D; Tue, 8 Jan 2013 15:02:29 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [207.238.187.242]) by elvis.mu.org (Postfix) with ESMTPSA id F06371A3D38; Tue, 8 Jan 2013 07:02:25 -0800 (PST) Message-ID: <50EC34FE.4070804@mu.org> Date: Tue, 08 Jan 2013 10:02:22 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130108145631.GD82219@kib.kiev.ua> In-Reply-To: <20130108145631.GD82219@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 15:02:30 -0000 On 1/8/13 9:56 AM, Konstantin Belousov wrote: > On Tue, Jan 08, 2013 at 01:09:51PM +0800, David Xu wrote: >> On 2013/01/08 02:22, Konstantin Belousov wrote: >>> Below is the forward of the patch for which I failed to obtain a private >>> review. Might be, the list generates more responses. >>> >>> Our rtld has a performance bootleneck, typically exposed by the images >>> with the lot of the run-time relocation processing, and by the C++ >>> exception handling. We block the signals delivery during the rtld >>> performing the relocations, as well as for the dl_iterate_phdr(3) (the >>> later is used for finding the dwarf unwinding tables). >>> >>> The signal blocking is needed to allow the rtld to resolve the symbols >>> for the signal handlers in the safe way, but also causes 2 syscalls >>> overhead per each rtld entry. >>> >>> The proposed approach allows to shave off those two syscalls, doubling >>> the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. >>> >>> Date: Mon, 13 Aug 2012 15:26:00 +0300 >>> From: Konstantin Belousov >>> >>> ... >>> >>> The basic idea is to implement sigprocmask() as single write into usermode >>> address. If kernel needs to calculate the signal mask for current thread, >>> it takes into the consideration non-zero value of the word at the agreed >>> address. Also, usermode is informed about signals which were put on hold >>> due to fast sigblock active. >>> >>> As I said, on my measurements in microbenchmark that did throw/catch in >>> a loop, I see equal user and system time spent for unpatched system, and >>> same user time with zero system time on patched system. >>> >>> Patch can be improved further, e.g. it would be nice to allow rtld to fall >>> back to sigprocmask(2) if kernel does not support fast sigblock, to prevent >>> flag day. Also, the mask enforced by fast sigblock can be made configurable. >>> >>> Note that libthr already blocks signals by catching them, and not using rtld >>> service in the first line handler. I tried to make the change in the spirit >>> of libthr interceptors, but handoff to libthr appears too complicated to >>> work. In fact, libthr can be changed to start using fast sigblock instead >>> of wrapping sigaction, but this is out of scope of the proposal right now. >>> >>> Please comment. >>> >>> diff --git a/lib/libc/sys/Symbol.map b/lib/libc/sys/Symbol.map >>> index 6888ea0..3b75539 100644 >>> --- a/lib/libc/sys/Symbol.map >>> +++ b/lib/libc/sys/Symbol.map >>> @@ -546,6 +546,7 @@ FBSDprivate_1.0 { >>> __sys_extattr_set_link; >>> _extattrctl; >>> __sys_extattrctl; >>> + __sys_fast_sigblock; >>> _fchdir; >>> __sys_fchdir; >>> _fchflags; >>> diff --git a/libexec/rtld-elf/rtld_lock.c b/libexec/rtld-elf/rtld_lock.c >>> index d1563e5..50c52c6 100644 >>> --- a/libexec/rtld-elf/rtld_lock.c >>> +++ b/libexec/rtld-elf/rtld_lock.c >>> @@ -43,6 +43,7 @@ >>> */ >>> >>> #include >>> +#include >>> #include >>> #include >>> #include >>> @@ -59,8 +60,8 @@ typedef struct Struct_Lock { >>> void *base; >>> } Lock; >>> >>> -static sigset_t fullsigmask, oldsigmask; >>> static int thread_flag; >>> +static uint32_t fsigblock; >>> >>> static void * >>> def_lock_create() >>> @@ -111,18 +112,26 @@ def_rlock_acquire(void *lock) >>> } >>> >>> static void >>> +sig_fastunblock(void) >>> +{ >>> + uint32_t oldval; >>> + >>> + oldval = atomic_fetchadd_32(&fsigblock, -FAST_SIGBLOCK_INC); >>> + if (oldval == (FAST_SIGBLOCK_PEND | FAST_SIGBLOCK_INC)) >>> + __sys_fast_sigblock(FAST_SIGBLOCK_UNBLOCK, NULL); >>> +} >>> + >>> +static void >>> def_wlock_acquire(void *lock) >>> { >>> Lock *l = (Lock *)lock; >>> - sigset_t tmp_oldsigmask; >>> >>> for ( ; ; ) { >>> - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); >>> + atomic_add_rel_32(&fsigblock, FAST_SIGBLOCK_INC); >>> if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) >>> break; >>> - sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); >>> + sig_fastunblock(); >>> } >>> - oldsigmask = tmp_oldsigmask; >>> } >>> >>> static void >>> @@ -134,7 +143,7 @@ def_lock_release(void *lock) >>> atomic_add_rel_int(&l->lock, -RC_INCR); >>> else { >>> atomic_add_rel_int(&l->lock, -WAFLAG); >>> - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); >>> + sig_fastunblock(); >>> } >>> } >>> >>> @@ -286,19 +295,7 @@ lockdflt_init() >>> >>> memcpy(&lockinfo, &deflockinfo, sizeof(lockinfo)); >>> _rtld_thread_init(NULL); >>> - /* >>> - * Construct a mask to block all signals except traps which might >>> - * conceivably be generated within the dynamic linker itself. >>> - */ >>> - sigfillset(&fullsigmask); >>> - sigdelset(&fullsigmask, SIGILL); >>> - sigdelset(&fullsigmask, SIGTRAP); >>> - sigdelset(&fullsigmask, SIGABRT); >>> - sigdelset(&fullsigmask, SIGEMT); >>> - sigdelset(&fullsigmask, SIGFPE); >>> - sigdelset(&fullsigmask, SIGBUS); >>> - sigdelset(&fullsigmask, SIGSEGV); >>> - sigdelset(&fullsigmask, SIGSYS); >>> + __sys_fast_sigblock(FAST_SIGBLOCK_SETPTR, &fsigblock); >>> } >>> >>> /* >>> @@ -319,7 +316,10 @@ _rtld_thread_init(struct RtldLockInfo *pli) >>> >>> if (pli == NULL) >>> pli = &deflockinfo; >>> - >>> + else { >>> + fsigblock = 0; >>> + __sys_fast_sigblock(FAST_SIGBLOCK_UNSETPTR, NULL); >>> + } >>> >>> for (i = 0; i < RTLD_LOCK_CNT; i++) >>> if ((locks[i] = pli->lock_create()) == NULL) >>> diff --git a/sys/compat/freebsd32/syscalls.master b/sys/compat/freebsd32/syscalls.master >>> index 0891e41..f9e8b9e 100644 >>> --- a/sys/compat/freebsd32/syscalls.master >>> +++ b/sys/compat/freebsd32/syscalls.master >>> @@ -997,3 +997,4 @@ >>> uint32_t offset1, uint32_t offset2,\ >>> uint32_t len1, uint32_t len2, \ >>> int advice); } >>> +532 AUE_NULL NOPROTO { int fast_sigblock(int cmd, uint32_t *ptr); } >>> diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c >>> index 90f7311..8a3cd15 100644 >>> --- a/sys/kern/kern_exec.c >>> +++ b/sys/kern/kern_exec.c >>> @@ -1031,6 +1031,7 @@ exec_new_vmspace(imgp, sv) >>> int error; >>> struct proc *p = imgp->proc; >>> struct vmspace *vmspace = p->p_vmspace; >>> + struct thread *td = curthread; >>> vm_object_t obj; >>> vm_offset_t sv_minuser, stack_addr; >>> vm_map_t map; >>> @@ -1039,6 +1040,10 @@ exec_new_vmspace(imgp, sv) >>> imgp->vmspace_destroyed = 1; >>> imgp->sysent = sv; >>> >>> + td->td_pflags &= ~TDP_FAST_SIGBLOCK; >>> + td->td_sigblock_ptr = NULL; >>> + td->td_sigblock_val = 0; >>> + >>> /* May be called with Giant held */ >>> EVENTHANDLER_INVOKE(process_exec, p, imgp); >>> >>> diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c >>> index 2685a8b..3c8a2f7 100644 >>> --- a/sys/kern/kern_sig.c >>> +++ b/sys/kern/kern_sig.c >>> @@ -230,6 +230,7 @@ static int sigproptbl[NSIG] = { >>> }; >>> >>> static void reschedule_signals(struct proc *p, sigset_t block, int flags); >>> +static sigset_t fastblock_mask; >>> >>> static void >>> sigqueue_start(void) >>> @@ -240,6 +241,16 @@ sigqueue_start(void) >>> p31b_setcfg(CTL_P1003_1B_REALTIME_SIGNALS, _POSIX_REALTIME_SIGNALS); >>> p31b_setcfg(CTL_P1003_1B_RTSIG_MAX, SIGRTMAX - SIGRTMIN + 1); >>> p31b_setcfg(CTL_P1003_1B_SIGQUEUE_MAX, max_pending_per_proc); >>> + SIGFILLSET(fastblock_mask); >>> + SIG_CANTMASK(fastblock_mask); >>> + SIGDELSET(fastblock_mask, SIGILL); >>> + SIGDELSET(fastblock_mask, SIGTRAP); >>> + SIGDELSET(fastblock_mask, SIGABRT); >>> + SIGDELSET(fastblock_mask, SIGEMT); >>> + SIGDELSET(fastblock_mask, SIGFPE); >>> + SIGDELSET(fastblock_mask, SIGBUS); >>> + SIGDELSET(fastblock_mask, SIGSEGV); >>> + SIGDELSET(fastblock_mask, SIGSYS); >>> } >>> >>> ksiginfo_t * >>> @@ -2525,6 +2536,7 @@ issignal(struct thread *td, int stop_allowed) >>> struct sigqueue *queue; >>> sigset_t sigpending; >>> int sig, prop, newsig; >>> + uint32_t oldval; >>> >>> p = td->td_proc; >>> ps = p->p_sigacts; >>> @@ -2541,6 +2553,32 @@ issignal(struct thread *td, int stop_allowed) >>> SIG_STOPSIGMASK(sigpending); >>> if (SIGISEMPTY(sigpending)) /* no signal to send */ >>> return (0); >>> + >>> + /* >>> + * Do fast sigblock if requested by usermode. Since >>> + * we do know that there was a signal pending at this >>> + * point, set the FAST_SIGBLOCK_PEND as indicator for >>> + * usermode to perform a dummy call to >>> + * FAST_SIGBLOCK_UNBLOCK, which causes immediate >>> + * delivery of postponed pending signal. >>> + */ >>> + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { >>> + if (td->td_sigblock_val != 0) >>> + SIGSETNAND(sigpending, fastblock_mask); >>> + if (SIGISEMPTY(sigpending)) { >>> + oldval = fuword32(td->td_sigblock_ptr); >>> + if (oldval == -1) { >>> + fetch_fast_sigblock_failed(td, 0); >>> + return (0); >>> + } >>> + oldval |= FAST_SIGBLOCK_PEND; >>> + if (suword32(td->td_sigblock_ptr, oldval) == -1) >>> + fetch_fast_sigblock_failed(td, 1); >>> + td->td_sigblock_val = oldval; >>> + return (0); >>> + } >>> + } >>> + >>> sig = sig_ffs(&sigpending); >>> >>> if (p->p_stops & S_SIG) { >>> @@ -3456,3 +3494,92 @@ sigacts_shared(struct sigacts *ps) >>> mtx_unlock(&ps->ps_mtx); >>> return (shared); >>> } >>> + >>> +int >>> +sys_fast_sigblock(struct thread *td, struct fast_sigblock_args *uap) >>> +{ >>> + struct proc *p; >>> + int error; >>> + uint32_t oldval; >>> + >>> + error = 0; >>> + switch (uap->cmd) { >>> + case FAST_SIGBLOCK_SETPTR: >>> + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) >>> + error = EBUSY; >>> + else if (((uintptr_t)(uap->ptr) & (sizeof(uint32_t) - 1)) != 0) >>> + error = EINVAL; >>> + else { >>> + td->td_pflags |= TDP_FAST_SIGBLOCK; >>> + td->td_sigblock_ptr = uap->ptr; >>> + } >>> + break; >>> + >>> + case FAST_SIGBLOCK_UNBLOCK: >>> + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { >>> + oldval = casuword32(td->td_sigblock_ptr, >>> + FAST_SIGBLOCK_PEND, 0); >>> + if (oldval == (uint32_t)-1) >>> + error = EFAULT; >>> + else if (oldval != FAST_SIGBLOCK_PEND) >>> + error = EBUSY; >>> + else >>> + td->td_sigblock_val = 0; >>> + } else >>> + error = EDOOFUS; >>> + >>> + /* >>> + * Rely on normal ast mechanism to deliver pending >>> + * signals to current thread. But notify others about >>> + * fake unblock. >>> + */ >>> + p = td->td_proc; >>> + if (error == 0 && p->p_numthreads != 1) { >>> + PROC_LOCK(p); >>> + reschedule_signals(p, td->td_sigmask, 0); >>> + PROC_UNLOCK(p); >>> + } >>> + break; >>> + >>> + case FAST_SIGBLOCK_UNSETPTR: >>> + if ((td->td_pflags & TDP_FAST_SIGBLOCK) != 0) { >>> + error = copyin(td->td_sigblock_ptr, &oldval, >>> + sizeof(uint32_t)); >>> + if (error != 0) >>> + break; >>> + if (oldval != 0 && oldval != FAST_SIGBLOCK_PEND) >>> + error = EBUSY; >>> + else { >>> + td->td_pflags &= ~TDP_FAST_SIGBLOCK; >>> + td->td_sigblock_val = 0; >>> + } >>> + } else >>> + error = EDOOFUS; >>> + break; >>> + } >>> + return (error); >>> +} >>> + >>> +void >>> +fetch_fast_sigblock(struct thread *td) >>> +{ >>> + >>> + if ((td->td_pflags & TDP_FAST_SIGBLOCK) == 0) >>> + return; >>> + td->td_sigblock_val = fuword32(td->td_sigblock_ptr); >>> + if (td->td_sigblock_val == -1) >>> + fetch_fast_sigblock_failed(td, 0); >>> +} >>> + >>> +void >>> +fetch_fast_sigblock_failed(struct thread *td, int write) >>> +{ >>> + ksiginfo_t ksi; >>> + >>> + td->td_sigblock_val = 0; >>> + ksiginfo_init_trap(&ksi); >>> + ksi.ksi_signo = SIGSEGV; >>> + ksi.ksi_code = write ? SEGV_ACCERR : SEGV_MAPERR; >>> + ksi.ksi_addr = td->td_sigblock_ptr; >>> + trapsignal(td, &ksi); >>> +} >>> diff --git a/sys/kern/subr_syscall.c b/sys/kern/subr_syscall.c >>> index 5aee684..77b250d 100644 >>> --- a/sys/kern/subr_syscall.c >>> +++ b/sys/kern/subr_syscall.c >>> @@ -131,6 +131,12 @@ syscallenter(struct thread *td, struct syscall_args *sa) >>> sa->callp, sa->args, 0); >>> #endif >>> >>> + /* >>> + * Fetch fast sigblock value at the time of syscall >>> + * entry because sleepqueue primitives might call >>> + * cursig(). >>> + */ >>> + fetch_fast_sigblock(td); >>> AUDIT_SYSCALL_ENTER(sa->code, td); >>> error = (sa->callp->sy_call)(td, sa->args); >>> AUDIT_SYSCALL_EXIT(error, td); >>> diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c >>> index 095bbdf..66e485b 100644 >>> --- a/sys/kern/subr_trap.c >>> +++ b/sys/kern/subr_trap.c >>> @@ -237,6 +237,7 @@ ast(struct trapframe *framep) >>> */ >>> if (flags & TDF_NEEDSIGCHK || p->p_pendingcnt > 0 || >>> !SIGISEMPTY(p->p_siglist)) { >>> + fetch_fast_sigblock(td); >>> PROC_LOCK(p); >>> mtx_lock(&p->p_sigacts->ps_mtx); >>> while ((sig = cursig(td, SIG_STOP_ALLOWED)) != 0) >>> diff --git a/sys/kern/syscalls.master b/sys/kern/syscalls.master >>> index f62dad7..28a9393 100644 >>> --- a/sys/kern/syscalls.master >>> +++ b/sys/kern/syscalls.master >>> @@ -951,5 +951,6 @@ >>> off_t offset, off_t len); } >>> 531 AUE_NULL STD { int posix_fadvise(int fd, off_t offset, \ >>> off_t len, int advice); } >>> +532 AUE_NULL STD { int fast_sigblock(int cmd, uint32_t *ptr); } >>> ; Please copy any additions and changes to the following compatability tables: >>> ; sys/compat/freebsd32/syscalls.master >>> diff --git a/sys/sys/proc.h b/sys/sys/proc.h >>> index 06df632..4899ca2 100644 >>> --- a/sys/sys/proc.h >>> +++ b/sys/sys/proc.h >>> @@ -272,6 +272,9 @@ struct thread { >>> struct osd td_osd; /* (k) Object specific data. */ >>> struct vm_map_entry *td_map_def_user; /* (k) Deferred entries. */ >>> pid_t td_dbg_forked; /* (c) Child pid for debugger. */ >>> + void *td_sigblock_ptr; /* (k) uptr for fast sigblock. */ >>> + uint32_t td_sigblock_val; /* (k) fast sigblock value at >>> + kernel entry. */ >>> #define td_endzero td_sigmask >>> >>> /* Copied during fork1() or create_thread(). */ >>> @@ -424,6 +427,7 @@ do { \ >>> #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */ >>> #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ >>> #define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */ >>> +#define TDP_FAST_SIGBLOCK 0x20000000 /* Fast sigblock active */ >>> >>> /* >>> * Reasons that the current thread can not be run yet. >>> diff --git a/sys/sys/signalvar.h b/sys/sys/signalvar.h >>> index 71685e7..68cca58 100644 >>> --- a/sys/sys/signalvar.h >>> +++ b/sys/sys/signalvar.h >>> @@ -250,6 +250,20 @@ typedef struct sigqueue { >>> /* Flags for ksi_flags */ >>> #define SQ_INIT 0x01 >>> >>> +/* >>> + * Fast_sigblock >>> + */ >>> +#define FAST_SIGBLOCK_SETPTR 1 >>> +#define FAST_SIGBLOCK_UNBLOCK 2 >>> +#define FAST_SIGBLOCK_UNSETPTR 3 >>> + >>> +#define FAST_SIGBLOCK_PEND 0x1 >>> +#define FAST_SIGBLOCK_INC 0x10 >>> + >>> +#ifndef _KERNEL >>> +int __sys_fast_sigblock(int cmd, void *ptr); >>> +#endif >>> + >>> #ifdef _KERNEL >>> >>> /* Return nonzero if process p has an unmasked pending signal. */ >>> @@ -329,6 +343,8 @@ extern struct mtx sigio_lock; >>> >>> int cursig(struct thread *td, int stop_allowed); >>> void execsigs(struct proc *p); >>> +void fetch_fast_sigblock(struct thread *td); >>> +void fetch_fast_sigblock_failed(struct thread *td, int write); >>> void gsignal(int pgid, int sig, ksiginfo_t *ksi); >>> void killproc(struct proc *p, char *why); >>> ksiginfo_t * ksiginfo_alloc(int wait); >>> >>> >>> >>> ----- End forwarded message ----- >>> >> So you want to make kernel share data with userland. The only thing I >> don't like is it adds overhead to syscall, rumor said that our syscall >> is slow than others, this might not be a problem ? >> >> Long time ago, I proposed a schedctl() syscall to make kernel share >> data with userland, some old patches are still there: >> http://people.freebsd.org/~davidxu/schedctl/ >> >> Mine does not have such an overhead, but it has another one: >> memory page is allocated in kernel which can not be swapped out >> and can not be freed until process is exited, the page is doubly >> mapped into in kernel and userland, accessing the shared data >> in kernel has zero overhead though. > Initial version of the changes indeed used the remap of the user page into > the KVA. Besides the issues you described regarding the page being wired > for the whole life of the thread, as well as the one page frame in the KVA, > there were other non-trivial problems. It was mostly related to the fork(2) > copying the address spaces, but not limited too. The use of the direct > user address access appeared to be much less intrusive. > > I completely agree with your note about the additional memory access in > the syscall entry path. I do believe that it does not incur a overhead > in the real-world loads, but might make some skews in the microbenchmarks. > I did not noted any in the testing I did. > > Anyway, I am still not sure is it worth optimizing for the throw/catch > microbenchmark at ll. If general public consequence is yes, the patch > needs some work before being committable anyway, mostly to allow patched > rtld to work on the pre-patch kernels. > > Thank you for the reply. I think it would be great if it works, I just haven't had time to fully understand your more recent patch. If you feel it's safe and maybe do a few test programs to see what happens, that would be best. -Alfred From owner-freebsd-toolchain@FreeBSD.ORG Wed Jan 9 07:38:50 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 979B49BF; Wed, 9 Jan 2013 07:38:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 0EF967AF; Wed, 9 Jan 2013 07:38:49 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r097chMb015543; Wed, 9 Jan 2013 09:38:43 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r097chMb015543 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r097cgIJ015542; Wed, 9 Jan 2013 09:38:42 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 9 Jan 2013 09:38:42 +0200 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130109073841.GH2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130108145631.GD82219@kib.kiev.ua> <50EC34FE.4070804@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="PNpeiK4tTqhYOExY" Content-Disposition: inline In-Reply-To: <50EC34FE.4070804@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2013 07:38:50 -0000 --PNpeiK4tTqhYOExY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 08, 2013 at 10:02:22AM -0500, Alfred Perlstein wrote: > I think it would be great if it works, I just haven't had time to fully= =20 > understand your more recent patch. If you feel it's safe and maybe do a= =20 > few test programs to see what happens, that would be best. I revived the test programs I used during the development of the patch, see http://people.freebsd.org/~kib/misc/fast_sigblock.c http://people.freebsd.org/~kib/misc/fast_sigblock2.c Also, I adapted the code to the specific case you asked about, in http://people.freebsd.org/~kib/misc/fast_sigblock3.c When run from the shell, it reports 'Killed'. --PNpeiK4tTqhYOExY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ7R6BAAoJEJDCuSvBvK1BzNcP/Aw2npROk0aBG7KzwWyLEWaB 2kI1ABq4nLTuPjhXlBCYOp+NZTJbPMYJuC6p7D+rmWm80sUN4nbaFsNPzGrbglHZ qGwavL0YDvJsCJ3JRfwe2SNYRP4Y1Hfb3D27TQLnYQOw3j2gETqaZTLGs4NJ6X8D R7BIho6Tkx/W7JVgK7CkYUSYPe7YXAu9HfInIN9X9B8WT4NkcgWUKTZTNkF/veBE WA32QPtrcOYR3r1Pf3OOnejRAipl45XV+OaglemrIcTwMYAGszgMdpmTAVEhaxuj 6xSit+8K0aFaZi98vrbMygeZo7Ghpkt257Q8Q2Pn8R0SpKcb1FiNQ2uRcgKARYx1 NPytWMjMdNYeR+ddYdAKm5Q8ibN0EFTmHxJFt7YngmEQ9nbmfcJRArY4iIYyGAeR ECAJxl5xe9zq8a9hY8CYVQt4pmf05NvdR4yEcwZk6p1sNEdA1wTr4/7lVKQgNJEg 64ot+aLHD9+GnpTU3ITZVPOk19hLGCG2buBF/1XG/hex4MELFiam6z+JwKN3ZD7m 5Em/A7OdBwrfUNu/hhZZziryIFPWoN7prOFJIDlMHlP3hPalm5JjBUJEr2Mo6lk6 04KzeH26/7qBmpyQqyT3RUGg4tQ3dvQTRefqjJfi4ESNJM5A+lY9wwc2uYPu3gkF pUVK4yxRIuiz0QFHi5GB =hGET -----END PGP SIGNATURE----- --PNpeiK4tTqhYOExY-- From owner-freebsd-toolchain@FreeBSD.ORG Wed Jan 9 15:52:22 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 70EB36EA; Wed, 9 Jan 2013 15:52:22 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5F28F9CF; Wed, 9 Jan 2013 15:52:22 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [207.238.187.242]) by elvis.mu.org (Postfix) with ESMTPSA id C78F01A3DBF; Wed, 9 Jan 2013 07:52:20 -0800 (PST) Message-ID: <50ED9232.8020606@mu.org> Date: Wed, 09 Jan 2013 10:52:18 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> <20130108145631.GD82219@kib.kiev.ua> <50EC34FE.4070804@mu.org> <20130109073841.GH2561@kib.kiev.ua> In-Reply-To: <20130109073841.GH2561@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2013 15:52:22 -0000 On 1/9/13 2:38 AM, Konstantin Belousov wrote: > On Tue, Jan 08, 2013 at 10:02:22AM -0500, Alfred Perlstein wrote: >> I think it would be great if it works, I just haven't had time to fully >> understand your more recent patch. If you feel it's safe and maybe do a >> few test programs to see what happens, that would be best. > I revived the test programs I used during the development of the patch, see > http://people.freebsd.org/~kib/misc/fast_sigblock.c > http://people.freebsd.org/~kib/misc/fast_sigblock2.c > > Also, I adapted the code to the specific case you asked about, in > http://people.freebsd.org/~kib/misc/fast_sigblock3.c > When run from the shell, it reports 'Killed'. This looks very good, thank you for addressing my concerns. I think you should go forward with it then. -Alfred From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 09:55:11 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F3650AED; Fri, 11 Jan 2013 09:55:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 515A6FFF; Fri, 11 Jan 2013 09:55:10 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0B9sxiX045840; Fri, 11 Jan 2013 11:54:59 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0B9sxiX045840 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0B9sxTZ045839; Fri, 11 Jan 2013 11:54:59 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 11 Jan 2013 11:54:59 +0200 From: Konstantin Belousov To: arch@freebsd.org, toolchain@freebsd.org Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130111095459.GZ2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NSDslKtQVY7LeFYu" Content-Disposition: inline In-Reply-To: <20130107182235.GA65279@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 09:55:11 -0000 --NSDslKtQVY7LeFYu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 07, 2013 at 08:22:35PM +0200, Konstantin Belousov wrote: > Below is the forward of the patch for which I failed to obtain a private > review. Might be, the list generates more responses. >=20 > Our rtld has a performance bootleneck, typically exposed by the images > with the lot of the run-time relocation processing, and by the C++ > exception handling. We block the signals delivery during the rtld > performing the relocations, as well as for the dl_iterate_phdr(3) (the > later is used for finding the dwarf unwinding tables). >=20 > The signal blocking is needed to allow the rtld to resolve the symbols > for the signal handlers in the safe way, but also causes 2 syscalls > overhead per each rtld entry. >=20 > The proposed approach allows to shave off those two syscalls, doubling > the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. >=20 > Date: Mon, 13 Aug 2012 15:26:00 +0300 > From: Konstantin Belousov >=20 > ... >=20 > The basic idea is to implement sigprocmask() as single write into usermode > address. If kernel needs to calculate the signal mask for current thread, > it takes into the consideration non-zero value of the word at the agreed > address. Also, usermode is informed about signals which were put on hold > due to fast sigblock active. >=20 > As I said, on my measurements in microbenchmark that did throw/catch in > a loop, I see equal user and system time spent for unpatched system, and > same user time with zero system time on patched system. >=20 > Patch can be improved further, e.g. it would be nice to allow rtld to fall > back to sigprocmask(2) if kernel does not support fast sigblock, to preve= nt > flag day. Also, the mask enforced by fast sigblock can be made configurab= le. >=20 > Note that libthr already blocks signals by catching them, and not using r= tld > service in the first line handler. I tried to make the change in the spir= it > of libthr interceptors, but handoff to libthr appears too complicated to > work. In fact, libthr can be changed to start using fast sigblock instead > of wrapping sigaction, but this is out of scope of the proposal right now. >=20 > Please comment. So there were no overly negative comments, and thanks to Alfred and David for useful notes. The patch at http://people.freebsd.org/~kib/misc/rtld-sigblock.2.patch is the commit candidate. Now kernel informs rtld about supprt for fast_sigblock with new auxv flag. Rtld can operate on old (and possibly future) kernels without fast_sigblock, rtld checks the auxv for presence of the ELF_BSDF_FASTSIGBLK flag before use. --NSDslKtQVY7LeFYu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ7+FyAAoJEJDCuSvBvK1BsJYP/iWcrdR8vkZY7F0vNFXpBxjR ut2gpD7pkI4biwsZECz8HzyR6nOceyxENv7ufyVu9M2bkIvwb7mIIfgnjAkBzvik yRh/LOl7eY0GBJpYtykF3m9dp3s+hPvF+SBg1NjQD5x8snC8uIWlToNRxms24P9i 0sUn4KosmntfedrVlLgf2cuXR3QJzOs7h9A1O/RBjelcMVMB1E1SmatrrCqAVWhz EwwKBdOhtyOWtKXEY3tkcsjsjI5nlsUnBOs17mwO1vqID956xZmCQGXM1/sM8yWH 2TAzKa89l06E9ql11lM+z6gzQ6oJb587S8VRPXgwaZU4pIKbgq3jbz2uFcsQKUiz IONGZyZBhO2VrvJWccE/5uASnc24yMck4eHztkibm6SqyrG0qnUZLxNmL2bUQXiN yg+U/DRxXjBBCFWiDnD6U3x+JlVbv91vIsOtrOGbmsIpXeZJ8ggybDD1ejR7n8fN 7V9UXneHrqM2/j7xFwZElqL7DfALVYv5pSqe/n0kf0/RkOZk3GMo3qZLf48im1A+ 8+2Iwez9MZDeWlIAJJPyrIPitZppjLCOb2NJZKFznc/tTQVg8Ug3GM8EnlzvRbw/ XltAhJilvGXGrZrrqxLekFeV9UUc0rQldZ/Wel9xFmrS5DrYNd94PME/lbRdzrEY ilQiq6EeRxZ893rISj3Q =66cr -----END PGP SIGNATURE----- --NSDslKtQVY7LeFYu-- From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 10:23:17 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id ABBC4973; Fri, 11 Jan 2013 10:23:17 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 833211C7; Fri, 11 Jan 2013 10:23:17 +0000 (UTC) Received: from xyf.my.dom (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r0BANGKh059347; Fri, 11 Jan 2013 10:23:16 GMT (envelope-from davidxu@freebsd.org) Message-ID: <50EFE830.3030500@freebsd.org> Date: Fri, 11 Jan 2013 18:23:44 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:14.0) Gecko/20120822 Thunderbird/14.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> In-Reply-To: <20130111095459.GZ2561@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 10:23:17 -0000 On 2013/01/11 17:54, Konstantin Belousov wrote: > On Mon, Jan 07, 2013 at 08:22:35PM +0200, Konstantin Belousov wrote: >> Below is the forward of the patch for which I failed to obtain a private >> review. Might be, the list generates more responses. >> >> Our rtld has a performance bootleneck, typically exposed by the images >> with the lot of the run-time relocation processing, and by the C++ >> exception handling. We block the signals delivery during the rtld >> performing the relocations, as well as for the dl_iterate_phdr(3) (the >> later is used for finding the dwarf unwinding tables). >> >> The signal blocking is needed to allow the rtld to resolve the symbols >> for the signal handlers in the safe way, but also causes 2 syscalls >> overhead per each rtld entry. >> >> The proposed approach allows to shave off those two syscalls, doubling >> the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. >> >> Date: Mon, 13 Aug 2012 15:26:00 +0300 >> From: Konstantin Belousov >> >> ... >> >> The basic idea is to implement sigprocmask() as single write into usermode >> address. If kernel needs to calculate the signal mask for current thread, >> it takes into the consideration non-zero value of the word at the agreed >> address. Also, usermode is informed about signals which were put on hold >> due to fast sigblock active. >> >> As I said, on my measurements in microbenchmark that did throw/catch in >> a loop, I see equal user and system time spent for unpatched system, and >> same user time with zero system time on patched system. >> >> Patch can be improved further, e.g. it would be nice to allow rtld to fall >> back to sigprocmask(2) if kernel does not support fast sigblock, to prevent >> flag day. Also, the mask enforced by fast sigblock can be made configurable. >> >> Note that libthr already blocks signals by catching them, and not using rtld >> service in the first line handler. I tried to make the change in the spirit >> of libthr interceptors, but handoff to libthr appears too complicated to >> work. In fact, libthr can be changed to start using fast sigblock instead >> of wrapping sigaction, but this is out of scope of the proposal right now. >> >> Please comment. > > So there were no overly negative comments, and thanks to Alfred and David > for useful notes. > > The patch at > http://people.freebsd.org/~kib/misc/rtld-sigblock.2.patch > is the commit candidate. Now kernel informs rtld about supprt for > fast_sigblock with new auxv flag. Rtld can operate on old (and possibly > future) kernels without fast_sigblock, rtld checks the auxv for > presence of the ELF_BSDF_FASTSIGBLK flag before use. > The patch looks fine to me. Sorry, I still have a question: if I found a process does not responsing to a signal, now should we check both its signal mask and fast block value ? but is there such a tool available ? for example, in gdb. From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 20:49:53 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 872CF55A; Fri, 11 Jan 2013 20:49:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D7A23CD; Fri, 11 Jan 2013 20:49:52 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0BKndIO014116; Fri, 11 Jan 2013 22:49:39 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0BKndIO014116 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0BKnc8c014115; Fri, 11 Jan 2013 22:49:38 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 11 Jan 2013 22:49:38 +0200 From: Konstantin Belousov To: David Xu Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130111204938.GD2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jYUWvSWTDpDT74zJ" Content-Disposition: inline In-Reply-To: <50EFE830.3030500@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 20:49:53 -0000 --jYUWvSWTDpDT74zJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jan 11, 2013 at 06:23:44PM +0800, David Xu wrote: > On 2013/01/11 17:54, Konstantin Belousov wrote: > > On Mon, Jan 07, 2013 at 08:22:35PM +0200, Konstantin Belousov wrote: > >> Below is the forward of the patch for which I failed to obtain a priva= te > >> review. Might be, the list generates more responses. > >> > >> Our rtld has a performance bootleneck, typically exposed by the images > >> with the lot of the run-time relocation processing, and by the C++ > >> exception handling. We block the signals delivery during the rtld > >> performing the relocations, as well as for the dl_iterate_phdr(3) (the > >> later is used for finding the dwarf unwinding tables). > >> > >> The signal blocking is needed to allow the rtld to resolve the symbols > >> for the signal handlers in the safe way, but also causes 2 syscalls > >> overhead per each rtld entry. > >> > >> The proposed approach allows to shave off those two syscalls, doubling > >> the FreeBSD performance for the (silly) throw/catch C++ microbenchmark. > >> > >> Date: Mon, 13 Aug 2012 15:26:00 +0300 > >> From: Konstantin Belousov > >> > >> ... > >> > >> The basic idea is to implement sigprocmask() as single write into user= mode > >> address. If kernel needs to calculate the signal mask for current thre= ad, > >> it takes into the consideration non-zero value of the word at the agre= ed > >> address. Also, usermode is informed about signals which were put on ho= ld > >> due to fast sigblock active. > >> > >> As I said, on my measurements in microbenchmark that did throw/catch in > >> a loop, I see equal user and system time spent for unpatched system, a= nd > >> same user time with zero system time on patched system. > >> > >> Patch can be improved further, e.g. it would be nice to allow rtld to = fall > >> back to sigprocmask(2) if kernel does not support fast sigblock, to pr= event > >> flag day. Also, the mask enforced by fast sigblock can be made configu= rable. > >> > >> Note that libthr already blocks signals by catching them, and not usin= g rtld > >> service in the first line handler. I tried to make the change in the s= pirit > >> of libthr interceptors, but handoff to libthr appears too complicated = to > >> work. In fact, libthr can be changed to start using fast sigblock inst= ead > >> of wrapping sigaction, but this is out of scope of the proposal right = now. > >> > >> Please comment. > > > > So there were no overly negative comments, and thanks to Alfred and Dav= id > > for useful notes. > > > > The patch at > > http://people.freebsd.org/~kib/misc/rtld-sigblock.2.patch > > is the commit candidate. Now kernel informs rtld about supprt for > > fast_sigblock with new auxv flag. Rtld can operate on old (and possibly > > future) kernels without fast_sigblock, rtld checks the auxv for > > presence of the ELF_BSDF_FASTSIGBLK flag before use. > > >=20 > The patch looks fine to me. Sorry, I still have a question: > if I found a process does not responsing to a signal, now should we > check both its signal mask and fast block value ? but is there such > a tool available ? for example, in gdb. Fair enough. I added the facility to export the address of the fast sigblock word to the tools, and implemented the procstat extension to print the address with -j. You can indeed use gdb to look up the content of the word when debugging, after getting it address from procstat. http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch --jYUWvSWTDpDT74zJ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8HriAAoJEJDCuSvBvK1BI/wP/i56RdjaKBiS48mkS24o2YHr yQ+NCpZE6Wj1PI37MnL/gPOPZyqoGEdsqdhDRfG0+dhEf2X1NCnOhWWHIaq4V2BJ CDhvpqLLU/Fd3N9N/JmaZvrHpZF7Bsvs36PLCAD7EChoX4KTnC9lEUqsfa9YUNuR 9eT7DmfSqGAcfUTlIs2EbudDI/EWkhXdx6PVAKGd6lBW74UQH6yeLrFJoB5ly6b9 dVEbufpcqy1OxbniM0BX9UgZUmbb3d3bE/tgrS/wrxo1gh4iMu5a2Ll/8zS6+PZm kcJ6pvp+4rTdmUxrgfmSlwgN+NIQpXGFOWtTR4zgyjOgK2uUkbUd9qujBHwmLoFS fJ7UIQCCyra3+GFPoEuRaHdrCMvEoR10J6zEHWN6TceIfei7FY0Qb2fbfRtW1If5 4p1tWoL0D+3lmO8aitdWQNIGEir4qXHQMFYW6zIBJC2qV8eMvxLW8uyCnq809vd7 5dmmwkybRR8n5vJQ2dXZwiWPF/K6o0WF+wI8bGMTAa4TO/TdKsy6AW1AtHOv33nO 8NChYxPM3s3YmjjKwp0qfrhHw7zacrooBPR3sHGpoDEMHSHsQESk5yOy0o63vSqG jJWAus44Lij0aQOCy4KJmkE7K41g0bUuUumSW4+D0+noDegykndojE5ulvj7RmPN z2GrZFFWTZd3O9NIbo7m =1Olc -----END PGP SIGNATURE----- --jYUWvSWTDpDT74zJ-- From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 20:52:10 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AADB47DA; Fri, 11 Jan 2013 20:52:10 +0000 (UTC) (envelope-from lev@serebryakov.spb.ru) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 59DBBFE; Fri, 11 Jan 2013 20:52:10 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:a15f:ca89:8194:fe2e]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 5D8A64AC2D; Sat, 12 Jan 2013 00:52:02 +0400 (MSK) Date: Sat, 12 Jan 2013 00:51:56 +0400 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <1533659601.20130112005156@serebryakov.spb.ru> To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) In-Reply-To: <20130111095459.GZ2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 20:52:10 -0000 Hello, Konstantin. You wrote 11 =D1=8F=D0=BD=D0=B2=D0=B0=D1=80=D1=8F 2013 =D0=B3., 13:54:59: KB> http://people.freebsd.org/~kib/misc/rtld-sigblock.2.patch KB> is the commit candidate. Now kernel informs rtld about supprt for Is it first patch in this state, which implements idea of shared memory between kernel and userland to spped up some syscalls? There were discussions about this approach for speedup of getpid()/getppid()/gettimeofday(), but without code ready for commit, am I right? --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 21:15:29 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C60FE16; Fri, 11 Jan 2013 21:15:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 990AF1D8; Fri, 11 Jan 2013 21:15:28 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0BLFMos017352; Fri, 11 Jan 2013 23:15:22 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0BLFMos017352 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0BLFMbi017351; Fri, 11 Jan 2013 23:15:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 11 Jan 2013 23:15:22 +0200 From: Konstantin Belousov To: Lev Serebryakov Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130111211522.GG2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <1533659601.20130112005156@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OWym5pP/pDvoNOpg" Content-Disposition: inline In-Reply-To: <1533659601.20130112005156@serebryakov.spb.ru> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 21:15:29 -0000 --OWym5pP/pDvoNOpg Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 12:51:56AM +0400, Lev Serebryakov wrote: > Hello, Konstantin. > You wrote 11 =D1=CE=D7=C1=D2=D1 2013 =C7., 13:54:59: >=20 >=20 > KB> http://people.freebsd.org/~kib/misc/rtld-sigblock.2.patch > KB> is the commit candidate. Now kernel informs rtld about supprt for > Is it first patch in this state, which implements idea of shared > memory between kernel and userland to spped up some syscalls? There > were discussions about this approach for speedup of > getpid()/getppid()/gettimeofday(), but without code ready for commit, > am I right? No, you are wrong. FreeBSD have shared page for very long time, and it is used for syscall-less gettimeofday for quite some time. Non-shared page which could carry per-thread or per-process information, neccessary for looking good in the trivial benchmarks, like getpid(), is different case. The patch does not implement the idea of speeding up some syscalls at all. It provides the special purpose one-off implementation of the alternate mechanism, for the internal consumption of rtld. --OWym5pP/pDvoNOpg Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8IDoAAoJEJDCuSvBvK1BZm8P/jfGlOBCq+iVPm29a0Uz4vW/ ssRW030DgmxZJkTALoH2Cq1HiU5jc2DmSqQL80UufQ97vKO3zWcIpU/iP3LbKuSF C+g+YOMPwbGFwAq2zWuSbQ8l8WNgxNivbWdiqR8UJguiZRJNm3ea88RXoRVGzOqV p1TwM4K/pEuyGqNGvUorZwAhVAI2d5YCL+UapGKFUX2hipg3G/F4Ruddnjn7Iwxi st8Bn+BnuIO/baHzIfleSC0xOaShnVOTv6bRTIxl8iKi7bzdZQ/0NEZSzl+v/yly mH18PxZB9s3Q/dAJ/HBHV0qwP4d59HvICWT078Pa/ePTqVqHjktOvQv7aG40WcH1 cWoZcKkjn0kzUk3iv9bu0G4p7an3AYhocwzBfWgyRKQoK8upzQBy2AFcA3jhEzLl qsjUzyBO2tMi4VxViySJr9rf8ilSDQDAG3GZtc0cG9XdxN93Z1et0v74J6FQZqt6 m2aSbkUtks4VqiPrC6Owf3DSdlWHKvIh36FnxnNWn2Vz2DvvEUm+sFCE5I3HwCU2 EutouzVdn+o167trHFHgASUBMdELShGNLh6ThN+fqVd9jn5b6CiBT18iMYf5JPU4 m88t1Gm/jN1RqwNnGV9TvlJXeefvUxWqZHEQb4phlILAae6z2k02x2wzB6rH2KKH HyQozZBs9LriINdOXz1A =sm/a -----END PGP SIGNATURE----- --OWym5pP/pDvoNOpg-- From owner-freebsd-toolchain@FreeBSD.ORG Fri Jan 11 23:29:21 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F2CF016D; Fri, 11 Jan 2013 23:29:20 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (unknown [IPv6:2001:610:1108:5012::107]) by mx1.freebsd.org (Postfix) with ESMTP id B370093F; Fri, 11 Jan 2013 23:29:20 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id C402F1203DE; Sat, 12 Jan 2013 00:29:06 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id A4F582848C; Sat, 12 Jan 2013 00:29:06 +0100 (CET) Date: Sat, 12 Jan 2013 00:29:06 +0100 From: Jilles Tjoelker To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130111232906.GA29017@stack.nl> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130111204938.GD2561@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 23:29:21 -0000 On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: > http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch The new fields td_sigblock_ptr and td_sigblock_val are in the part that is zeroed for new threads, while the code in rtld appears to expect them to be copied (on fork, vfork and pthread_create). The fields are correctly zeroed on exec. Sharing the magic variable between threads means that one thread holding an rtld lock will block signals for all other threads as well. This is different from how the normal signal mask works but I don't know whether it may break things. However, the patch depends on it in some way because sigtd() does not know about fast sigblock and will therefore happily select a thread that is fast-sigblocking to handle a signal directed at the process. Although libthr's postpone approach is somewhat ugly, it does not depend on any non-standard kernel features and does not delay the default action. Would it be possible to move that code to libc to make things easier for rtld? It looks like this requires teaching libc about various threading concepts, though. Something feels ugly about not allowing applications to use this, although I don't know how it would work. On the other hand, the strange signal state created by this and implicit assumptions that the duration will be short may be a reason to restrict its use to a relatively small piece of code. -- Jilles Tjoelker From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 00:27:12 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 886372C5; Sat, 12 Jan 2013 00:27:12 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from mail-gg0-f173.google.com (mail-gg0-f173.google.com [209.85.161.173]) by mx1.freebsd.org (Postfix) with ESMTP id 27BBEB43; Sat, 12 Jan 2013 00:27:11 +0000 (UTC) Received: by mail-gg0-f173.google.com with SMTP id f2so437687ggn.18 for ; Fri, 11 Jan 2013 16:27:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:reply-to:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=rPSsYlKDdjJKAN6OxfLjsSv8ldhkvqciiCO6YXHO3jk=; b=kijs77MHQYJHyA6ISRj0q32xVUtucxtq8AfSGbr4UektWEKwIwh/80m0iKi9OXdODH CT6rp922iHv1T5KO2uArsB2IeZUWdPAACGE7DZBOvkqpdNruw2m23XVZJ/hFXVFLWGAK aOdW58wwOq+8kQT/6qKMdrWEV/r9/n75seqVyAGA5fqXs5m4qGzWkdq0FwOHO4evlHh+ W4bFCJhDqh/m41qmIHhgu1G3baLkYXAk5iERCXK2RiLYfxXKRz72czKNAB1LDHP14G7D VnZKNzIDQ8JaRI9H6OjNlAmbh4PAHXru7hctA8SILcmbMpWWzCDPQ5USR2ZoCZCC3qtf rwUQ== X-Received: by 10.236.93.69 with SMTP id k45mr11007395yhf.28.1357950431277; Fri, 11 Jan 2013 16:27:11 -0800 (PST) Received: from xp5k.my.domain ([115.192.135.231]) by mx.google.com with ESMTPS id r9sm5910613yhm.14.2013.01.11.16.27.08 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 11 Jan 2013 16:27:10 -0800 (PST) Message-ID: <50F0ADDA.4000801@gmail.com> Date: Sat, 12 Jan 2013 08:27:06 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Jilles Tjoelker Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> In-Reply-To: <20130111232906.GA29017@stack.nl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: davidxu@freebsd.org List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 00:27:12 -0000 On 2013/01/12 07:29, Jilles Tjoelker wrote: > On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: >> http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch > The new fields td_sigblock_ptr and td_sigblock_val are in the part that > is zeroed for new threads, while the code in rtld appears to expect them > to be copied (on fork, vfork and pthread_create). The fields are > correctly zeroed on exec. > > Sharing the magic variable between threads means that one thread holding > an rtld lock will block signals for all other threads as well. This is > different from how the normal signal mask works but I don't know whether > it may break things. However, the patch depends on it in some way > because sigtd() does not know about fast sigblock and will therefore > happily select a thread that is fast-sigblocking to handle a signal > directed at the process. > > Although libthr's postpone approach is somewhat ugly, it does not depend > on any non-standard kernel features and does not delay the default > action. Would it be possible to move that code to libc to make things > easier for rtld? It looks like this requires teaching libc about various > threading concepts, though. Long time ago, if i remembered correctly, kib said that he wanted to merge libthr code into libc, I don't know its state. > Something feels ugly about not allowing applications to use this, > although I don't know how it would work. On the other hand, the strange > signal state created by this and implicit assumptions that the duration > will be short may be a reason to restrict its use to a relatively small > piece of code. > True, it seems it is for short duration. From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 05:32:03 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 89561A79; Sat, 12 Jan 2013 05:32:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id DB09A655; Sat, 12 Jan 2013 05:32:02 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0C5VmH5033003; Sat, 12 Jan 2013 07:31:48 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0C5VmH5033003 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0C5VmRC033002; Sat, 12 Jan 2013 07:31:48 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 12 Jan 2013 07:31:48 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130112053147.GH2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1vEdgMWljt97yA53" Content-Disposition: inline In-Reply-To: <20130111232906.GA29017@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 05:32:03 -0000 --1vEdgMWljt97yA53 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 12:29:06AM +0100, Jilles Tjoelker wrote: > On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: > > http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch >=20 > The new fields td_sigblock_ptr and td_sigblock_val are in the part that > is zeroed for new threads, while the code in rtld appears to expect them > to be copied (on fork, vfork and pthread_create). The fields are > correctly zeroed on exec. Thank you for noting this. Should be fixed in http://people.freebsd.org/~kib/misc/rtld-sigblock.4.patch >=20 > Sharing the magic variable between threads means that one thread holding > an rtld lock will block signals for all other threads as well. This is > different from how the normal signal mask works but I don't know whether > it may break things. However, the patch depends on it in some way > because sigtd() does not know about fast sigblock and will therefore > happily select a thread that is fast-sigblocking to handle a signal > directed at the process. Hm, I do not quite follow, at least not the first part of the paragraph. The fast sigblock pointer is per-thread, so it is not shared in the kernel. Regardless of the kernel side, rtld is only supposed to use the fast block in the default implementation of the rtld locks, which are overriden by the libthr implementation on libthr initialization. There is also an explicit hand-off from the default locks to the external (libthr), and rtld cares to turn off fast sigblock mechanism when the handoff is performed. The selection of the thread for the delivery of signal in the mt process should not matter then, due to the mechanism not supposed to be used in the mt process. >=20 > Although libthr's postpone approach is somewhat ugly, it does not depend > on any non-standard kernel features and does not delay the default > action. Would it be possible to move that code to libc to make things > easier for rtld? It looks like this requires teaching libc about various > threading concepts, though. The concern there is that rtld would need to have a tight coupling with libc, and possibly with libthr too. >=20 > Something feels ugly about not allowing applications to use this, > although I don't know how it would work. On the other hand, the strange > signal state created by this and implicit assumptions that the duration > will be short may be a reason to restrict its use to a relatively small > piece of code. Application use of the interface is equivalent to the application willingly corrupt its own state. --1vEdgMWljt97yA53 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8PVDAAoJEJDCuSvBvK1BoRkQAI5otpa95LCidyeDwAW/TN2w R+VMZahZeIGNS8SWABgZnDqBfDHmwnIUDmg4Em/2z/yutPbBr74cl2h7JCCXLwaj FidmiaC18+O8AY68ir9uqt4QGt7VlNbcDutb9KSghdbFVVl0qF18OBNckv5jhltl sQu53uKng5x4DMP3yGxn/7icQVRr1koOwQJku1H7af1c39VIvpjbgLFBucfKlwVX LQkpk5EcjjvjL+SSTVzQJrSzvulj+bE2OXH+24QN+mClY07SjjAgHrASH/SHulAl UkN7H8XqoAEK7YYdz2lDedSOlYmkstykoFuoVbQhTvU+7PN23aEvNkOpi+UUFcZN ZHd45SiQ1/LHNC8W3CUnPzDNEcTMxkdYoNe0qKw43ZkaLgIrFVgM9Je9Ouy+QRCX ij/Y55X4UdqHhBG5dOGg/cvYgjkStAPEVGW+Of0L1YlGxvUGizVU82FTsSszBrH7 Qv1o13mhs1Mas/NXCaP37G4hSJzmYjMBEXcgTYSKb3po07uDErilBx/hkCl+EGBD wU5e25CTQT284fWMB8xxgx7FQ6c0gTLkWeTHf0TBYHEWyt3qT//GrhJoFOm4uCJ8 vgyt+sAbaPLg8rWt26kUZuHnVn3e+n51aPk4hIFBreCAt6mg4HH6OV1yDVPwmwHW Ri1zZ8wEM3SAvomdsBwN =clJL -----END PGP SIGNATURE----- --1vEdgMWljt97yA53-- From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 05:32:58 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DCAC8B37; Sat, 12 Jan 2013 05:32:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 7F03265D; Sat, 12 Jan 2013 05:32:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0C5Wq1Z033020; Sat, 12 Jan 2013 07:32:52 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0C5Wq1Z033020 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0C5WqBT033019; Sat, 12 Jan 2013 07:32:52 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 12 Jan 2013 07:32:52 +0200 From: Konstantin Belousov To: davidxu@freebsd.org Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130112053252.GI2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <50F0ADDA.4000801@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="O2lvLMQR9k8T/RyB" Content-Disposition: inline In-Reply-To: <50F0ADDA.4000801@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, Jilles Tjoelker , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 05:32:58 -0000 --O2lvLMQR9k8T/RyB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 08:27:06AM +0800, David Xu wrote: > On 2013/01/12 07:29, Jilles Tjoelker wrote: > > On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: > >> http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch > > The new fields td_sigblock_ptr and td_sigblock_val are in the part that > > is zeroed for new threads, while the code in rtld appears to expect them > > to be copied (on fork, vfork and pthread_create). The fields are > > correctly zeroed on exec. > > > > Sharing the magic variable between threads means that one thread holding > > an rtld lock will block signals for all other threads as well. This is > > different from how the normal signal mask works but I don't know whether > > it may break things. However, the patch depends on it in some way > > because sigtd() does not know about fast sigblock and will therefore > > happily select a thread that is fast-sigblocking to handle a signal > > directed at the process. > > > > Although libthr's postpone approach is somewhat ugly, it does not depend > > on any non-standard kernel features and does not delay the default > > action. Would it be possible to move that code to libc to make things > > easier for rtld? It looks like this requires teaching libc about various > > threading concepts, though. > Long time ago, if i remembered correctly, kib said that he wanted to > merge libthr code into libc, I don't know its state. Yes, this is correct, and I still want this. >=20 > > Something feels ugly about not allowing applications to use this, > > although I don't know how it would work. On the other hand, the strange > > signal state created by this and implicit assumptions that the duration > > will be short may be a reason to restrict its use to a relatively small > > piece of code. > > > True, it seems it is for short duration. --O2lvLMQR9k8T/RyB Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8PWDAAoJEJDCuSvBvK1BDgUP/iWJXQukG9XrXeguc+y+bAOf VcWtQfyWdFz0ZsOLdKH6z/gnRF/yiQynvp+na+rtq0rgWDBXIQ7uP0ZKItAevfkW nFM7s12Gv1bQNQq59uFeyshU97ORHCk11ZpjLs84mcKPDWEyAsZQAxRXhz5oTSSL SAkV3ooYeE6eSuFA6ZQgUT3WKDbTwpTTm1mTjGjnPWrRYNrKkrNNZ2P6E7uANawi GWMX0sMbrhtrrmrwUPNdR7ny8jqcaQrCvLxZ+73cVEQBIZIhEuHt9QxqEwZm+OYX v9xV4cSZfCLC2HA/zVxzch5HL+ZExBKDVkSx0K394giPzjlymJePq18VgiBTcozj 6tRPNaWNzUZj9ACwbXsr9sGPuv+/e3VqAR0b5nREOatqy2ha04Q80PZpyIf9Umwx 7D8qsRxmhQJhWfvPevaOeD22OuCy2Wv9qQ76p6OnprsNt8VMStMnBKoxpBL024Na 6VsUaK3QpI3zI1NLdzQyvbfXDd8pQBjYm6NC2Jv3ukX7uc1D3JXJtOESr1TnWyWe QwT8Nf1hLM24mtqJQRipYE4wYSEYXNK3A5v1eXCF3PG/VipQ/I4j+3WPOmDdS6xu E7y5hTNBlQvZuZIFb2FVcuvTuu3AgOB9hdmheZA/xJzsiHUy6dEYWrHST3FUhxHr d7lyOHIZ890gPcsouqy7 =YOKS -----END PGP SIGNATURE----- --O2lvLMQR9k8T/RyB-- From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 11:56:41 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 97BBD8E1; Sat, 12 Jan 2013 11:56:41 +0000 (UTC) (envelope-from theraven@theravensnest.org) Received: from theravensnest.org (theraven.freebsd.your.org [216.14.102.27]) by mx1.freebsd.org (Postfix) with ESMTP id 4810A2C0; Sat, 12 Jan 2013 11:56:40 +0000 (UTC) Received: from [192.168.0.2] (cpc10-cmbg15-2-0-cust123.5-4.cable.virginmedia.com [86.30.246.124]) (authenticated bits=0) by theravensnest.org (8.14.5/8.14.5) with ESMTP id r0CBuU1c001928 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Sat, 12 Jan 2013 11:56:33 GMT (envelope-from theraven@theravensnest.org) Subject: Re: LLVM Image Activator Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: David Chisnall In-Reply-To: <50E9BC2D.7000302@freebsd.org> Date: Sat, 12 Jan 2013 11:56:24 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: <7ADA6363-E714-42E7-82FA-BDE6C64C75A9@theravensnest.org> References: <50E9BC2D.7000302@freebsd.org> To: Nathan Whitehorn X-Mailer: Apple Mail (2.1278) Cc: freebsd-toolchain@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 11:56:41 -0000 Hi Nathan, I'm very interested in making better use of LLVM in our infrastructure. = It would also be nice to be able to integrate this with rtld, so that we = can load shared libraries that contain LLVM IR. Beyond that, there are a few long-term projects that I have in mind: - Run passes that add code for collecting profiling information before = running code invoked in this way. - Recompile on subsequent invocations based on the collected profiling = information. - Specialise installed shared libraries based on the programs that link = to them, so that if foo band bar both link to libbaz, but use it in very = different ways, they get different versions compiled specifically for = them. - Statically compile and cache programs invoked with the JIT, so that = they are very fast on their second run. This would probably want a = periodic job to remove old caches. =20 I have written code in LanguageKit for doing several of these things, = but it would be nicer to have them done at the system level, rather than = at the level of a front end. I'd also like to start using the LLVM JIT for accelerating various other = functions. I have a proof-of-concept JIT for BPF that uses LLVM and I'd = like to extend this to pf/ipf firewall rules. David On 6 Jan 2013, at 18:02, Nathan Whitehorn wrote: > Having LLVM/clang in the base system lets us do some interesting = things > that we couldn't do with GCC. One is that LLVM ships with a JIT for = LLVM > IR as well as components of a toolchain for it (this is what Google's > pNACL uses) and that you can end up producing binary files that are in > IR instead of native code. The IR isn't really cross-platform, but = does > let you do CPU-specific optimizations when executed by the JIT, etc. >=20 > The attached patch causes the LLVM JIT (lli) to be built by default > (adding ~20 seconds to buildworld on my five-year-old laptop) and adds = a > kernel image activator that invokes it when passed LLVM bitcode files. > It's not completely finished (see the XXX comment in the middle), but = it > does work, as follows: >=20 > $ clang -emit-llvm -c -o hw.ll hw.c > $ file hw.ll > hw.ll: LLVM bitcode > $ lli hw.ll > Hello world! > $ chmod a+x hw.ll > $ ./hw.ll > Hello world! > $ >=20 > Is there any interest in having features like this? It seems like this > could provides some interesting possibilities for us and nice > integration from having imported clang into base. > -Nathan > _______________________________________________ > freebsd-toolchain@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain > To unsubscribe, send any mail to = "freebsd-toolchain-unsubscribe@freebsd.org" From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 16:25:51 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5404CA58; Sat, 12 Jan 2013 16:25:51 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id DE92CEFF; Sat, 12 Jan 2013 16:25:50 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id F1529359313; Sat, 12 Jan 2013 17:25:47 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id DA5F62848C; Sat, 12 Jan 2013 17:25:47 +0100 (CET) Date: Sat, 12 Jan 2013 17:25:47 +0100 From: Jilles Tjoelker To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130112162547.GA54954@stack.nl> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <20130112053147.GH2561@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130112053147.GH2561@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 16:25:51 -0000 On Sat, Jan 12, 2013 at 07:31:48AM +0200, Konstantin Belousov wrote: > On Sat, Jan 12, 2013 at 12:29:06AM +0100, Jilles Tjoelker wrote: > > On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: > > > http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch > > The new fields td_sigblock_ptr and td_sigblock_val are in the part that > > is zeroed for new threads, while the code in rtld appears to expect them > > to be copied (on fork, vfork and pthread_create). The fields are > > correctly zeroed on exec. > Thank you for noting this. Should be fixed in > http://people.freebsd.org/~kib/misc/rtld-sigblock.4.patch > > Sharing the magic variable between threads means that one thread holding > > an rtld lock will block signals for all other threads as well. This is > > different from how the normal signal mask works but I don't know whether > > it may break things. However, the patch depends on it in some way > > because sigtd() does not know about fast sigblock and will therefore > > happily select a thread that is fast-sigblocking to handle a signal > > directed at the process. > Hm, I do not quite follow, at least not the first part of the paragraph. > The fast sigblock pointer is per-thread, so it is not shared in the kernel. > Regardless of the kernel side, rtld is only supposed to use the fast > block in the default implementation of the rtld locks, which are overriden > by the libthr implementation on libthr initialization. There is also an > explicit hand-off from the default locks to the external (libthr), and > rtld cares to turn off fast sigblock mechanism when the handoff is > performed. > The selection of the thread for the delivery of signal in the mt process > should not matter then, due to the mechanism not supposed to be used > in the mt process. OK, you are right. If multiple threads exist, the patched code disables fast sigblock and the current unpatched code already postpones signal handlers without sigprocmask() syscalls. This suggests a different rather simpler change. Libthr can always use its rtld lock implementation instead of only when multiple threads exist. This avoids the sigprocmask() syscalls and should not be much slower than the default implementation in rtld because that also contains atomic operations (both the unpatched and the patched version). People that care about performance of exceptions can then link in libthr (in many cases, it is already linked in to allow for (the possibility of) threading). I have tested this and exceptions were indeed more than twice as fast in my test program if I created an extra thread doing nothing than in the fully single-threaded version (with or without libthr). With that, I think fast sigblock is too much code and complication for a niche case. Most of the extra atomics in multi-threaded applications are conditional on __isthreaded (or can be made so); therefore, performance loss from linking in libthr should be negligible in most cases. > > Although libthr's postpone approach is somewhat ugly, it does not depend > > on any non-standard kernel features and does not delay the default > > action. Would it be possible to move that code to libc to make things > > easier for rtld? It looks like this requires teaching libc about various > > threading concepts, though. > The concern there is that rtld would need to have a tight coupling > with libc, and possibly with libthr too. Libc could call _rtld_thread_init() during initialization, somewhat like libthr does now. The problems here are in libc and libthr, not in rtld. Before libc is ready, the current code can be used; if libc is always initialized first, the sigprocmask() calls can be removed because there is no way a signal handler can be installed that early. -- Jilles Tjoelker From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 17:22:10 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BECBEB18; Sat, 12 Jan 2013 17:22:10 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 8AA40123; Sat, 12 Jan 2013 17:22:10 +0000 (UTC) Received: from JRE-MBP-2.local (c-50-143-148-105.hsd1.ca.comcast.net [50.143.148.105]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id r0CHM1JN020230 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 12 Jan 2013 09:22:01 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <50F19BB4.6080309@freebsd.org> Date: Sat, 12 Jan 2013 09:21:56 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fast sigblock (AKA rtld speedup) References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <20130112053147.GH2561@kib.kiev.ua> In-Reply-To: <20130112053147.GH2561@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, Jilles Tjoelker , toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 17:22:10 -0000 On 1/11/13 9:31 PM, Konstantin Belousov wrote: > On Sat, Jan 12, 2013 at 12:29:06AM +0100, Jilles Tjoelker wrote: >> On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: >>> http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch >> The new fields td_sigblock_ptr and td_sigblock_val are in the part that >> is zeroed for new threads, while the code in rtld appears to expect them >> to be copied (on fork, vfork and pthread_create). The fields are >> correctly zeroed on exec. > Thank you for noting this. Should be fixed in > http://people.freebsd.org/~kib/misc/rtld-sigblock.4.patch > >> Sharing the magic variable between threads means that one thread holding >> an rtld lock will block signals for all other threads as well. This is >> different from how the normal signal mask works but I don't know whether >> it may break things. However, the patch depends on it in some way >> because sigtd() does not know about fast sigblock and will therefore >> happily select a thread that is fast-sigblocking to handle a signal >> directed at the process. > Hm, I do not quite follow, at least not the first part of the paragraph. > > The fast sigblock pointer is per-thread, so it is not shared in the kernel. > Regardless of the kernel side, rtld is only supposed to use the fast > block in the default implementation of the rtld locks, which are overriden > by the libthr implementation on libthr initialization. There is also an > explicit hand-off from the default locks to the external (libthr), and > rtld cares to turn off fast sigblock mechanism when the handoff is > performed. I assume that the thread notifies the kernel of the location.. Might I suggest as a saftybelt that on notification of this address that the kernel requires that a known "magic" value be in this location as proof that all is as expected.? just a thought. > > The selection of the thread for the delivery of signal in the mt process > should not matter then, due to the mechanism not supposed to be used > in the mt process. >> Although libthr's postpone approach is somewhat ugly, it does not depend >> on any non-standard kernel features and does not delay the default >> action. Would it be possible to move that code to libc to make things >> easier for rtld? It looks like this requires teaching libc about various >> threading concepts, though. > The concern there is that rtld would need to have a tight coupling > with libc, and possibly with libthr too. > >> Something feels ugly about not allowing applications to use this, >> although I don't know how it would work. On the other hand, the strange >> signal state created by this and implicit assumptions that the duration >> will be short may be a reason to restrict its use to a relatively small >> piece of code. > Application use of the interface is equivalent to the application > willingly corrupt its own state. From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 21:16:48 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 95663E5B; Sat, 12 Jan 2013 21:16:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 09B7A9C6; Sat, 12 Jan 2013 21:16:47 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r0CLGe0e031517; Sat, 12 Jan 2013 23:16:40 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.4 kib.kiev.ua r0CLGe0e031517 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r0CLGelQ031516; Sat, 12 Jan 2013 23:16:40 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 12 Jan 2013 23:16:40 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130112211640.GN2561@kib.kiev.ua> References: <20130107182235.GA65279@kib.kiev.ua> <20130111095459.GZ2561@kib.kiev.ua> <50EFE830.3030500@freebsd.org> <20130111204938.GD2561@kib.kiev.ua> <20130111232906.GA29017@stack.nl> <20130112053147.GH2561@kib.kiev.ua> <20130112162547.GA54954@stack.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4cokgWgqjr3t8EL1" Content-Disposition: inline In-Reply-To: <20130112162547.GA54954@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 21:16:48 -0000 --4cokgWgqjr3t8EL1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 05:25:47PM +0100, Jilles Tjoelker wrote: > On Sat, Jan 12, 2013 at 07:31:48AM +0200, Konstantin Belousov wrote: > > On Sat, Jan 12, 2013 at 12:29:06AM +0100, Jilles Tjoelker wrote: > > > On Fri, Jan 11, 2013 at 10:49:38PM +0200, Konstantin Belousov wrote: > > > > http://people.freebsd.org/~kib/misc/rtld-sigblock.3.patch >=20 > > > The new fields td_sigblock_ptr and td_sigblock_val are in the part th= at > > > is zeroed for new threads, while the code in rtld appears to expect t= hem > > > to be copied (on fork, vfork and pthread_create). The fields are > > > correctly zeroed on exec. > > Thank you for noting this. Should be fixed in > > http://people.freebsd.org/~kib/misc/rtld-sigblock.4.patch >=20 > > > Sharing the magic variable between threads means that one thread hold= ing > > > an rtld lock will block signals for all other threads as well. This is > > > different from how the normal signal mask works but I don't know whet= her > > > it may break things. However, the patch depends on it in some way > > > because sigtd() does not know about fast sigblock and will therefore > > > happily select a thread that is fast-sigblocking to handle a signal > > > directed at the process. >=20 > > Hm, I do not quite follow, at least not the first part of the paragraph. >=20 > > The fast sigblock pointer is per-thread, so it is not shared in the ker= nel. > > Regardless of the kernel side, rtld is only supposed to use the fast > > block in the default implementation of the rtld locks, which are overri= den > > by the libthr implementation on libthr initialization. There is also an > > explicit hand-off from the default locks to the external (libthr), and > > rtld cares to turn off fast sigblock mechanism when the handoff is > > performed. >=20 > > The selection of the thread for the delivery of signal in the mt process > > should not matter then, due to the mechanism not supposed to be used > > in the mt process. >=20 > OK, you are right. If multiple threads exist, the patched code disables > fast sigblock and the current unpatched code already postpones signal > handlers without sigprocmask() syscalls. >=20 > This suggests a different rather simpler change. Libthr can always use > its rtld lock implementation instead of only when multiple threads > exist. This avoids the sigprocmask() syscalls and should not be much > slower than the default implementation in rtld because that also > contains atomic operations (both the unpatched and the patched version). > People that care about performance of exceptions can then link in libthr > (in many cases, it is already linked in to allow for (the possibility > of) threading). >=20 > I have tested this and exceptions were indeed more than twice as fast in > my test program if I created an extra thread doing nothing than in the > fully single-threaded version (with or without libthr). >=20 > With that, I think fast sigblock is too much code and complication for a > niche case. Ok, thank you for the review. I am reverting the branch. I think a solution would need to wait while the work to merge libc and libthr is done. >=20 > Most of the extra atomics in multi-threaded applications are conditional > on __isthreaded (or can be made so); therefore, performance loss from > linking in libthr should be negligible in most cases. >=20 > > > Although libthr's postpone approach is somewhat ugly, it does not dep= end > > > on any non-standard kernel features and does not delay the default > > > action. Would it be possible to move that code to libc to make things > > > easier for rtld? It looks like this requires teaching libc about vari= ous > > > threading concepts, though. >=20 > > The concern there is that rtld would need to have a tight coupling > > with libc, and possibly with libthr too. >=20 > Libc could call _rtld_thread_init() during initialization, somewhat like > libthr does now. The problems here are in libc and libthr, not in rtld. > Before libc is ready, the current code can be used; if libc is always > initialized first, the sigprocmask() calls can be removed because there > is no way a signal handler can be installed that early. >=20 > --=20 > Jilles Tjoelker --4cokgWgqjr3t8EL1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8dK2AAoJEJDCuSvBvK1BYkMP/i+lV09PclmW2T/9PjvEm8rZ E4xLnrt4S5tBt1OZLqlSnSRsZ4KUUg0PfOIq7YODHkE6YjaonUb2wVcxa5hPHST3 ea+7waOb0ifKIBgVZ6pkQtHG3IbqRimiJrxVIg7jnz+BSGgA1vpz6shWLT9+aV/A 524lTC6zUiRj9JyAYDlevQ2VTDMAW/BcYS8vI9+9Qmg4nIxsSYTWLcpOlr/iJGCd fxp431WOv+XYlSaknzx+fEBrelXEEEfgY1DbW3BjYH/zqZvceZvQN4fCWmSlJ/uf 5dS86FiykyJb1KqKvQy00esVHIwsMlmhgb4BSzsNuhBPZZTvxlA69FkUONP/s9jb ht8j2FeCaGE3He80IaORWJCAjpPbdeYlBM1Q3D0Rwz3R8zXrjHGbBMhgEKXvEcq3 ZPouvvsuMX6NYlLlCVpgxTOfxmdN4tdCXZolGyb7VAMi9bjCSVrpEJlflxJAv65t JUvJ1+kvcCb7/Bb+oAvC35SpUB7tMX4zIS+0S9XkJCAk/9JF5PgLvKrbN+aRK5Bz bqEIE2gSU4QfVOCigJv3QAt7ZIFpGiRZe5mkcKximPuVaQRBNFKzrt1so2hxfuHE 5Sjsit8zujDlNGejtLM5/Yl+uJYoNick+icMk7/8ubCzQyPyS152ZkfOfax4S8// PI1LW0Kv6qMyukPKjJcA =FUvI -----END PGP SIGNATURE----- --4cokgWgqjr3t8EL1-- From owner-freebsd-toolchain@FreeBSD.ORG Sat Jan 12 23:04:42 2013 Return-Path: Delivered-To: toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0DA1A579; Sat, 12 Jan 2013 23:04:42 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id C9C91CF3; Sat, 12 Jan 2013 23:04:41 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r0CN4ZVb081699 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 12 Jan 2013 15:04:35 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r0CN4ZSM081698; Sat, 12 Jan 2013 15:04:35 -0800 (PST) (envelope-from jmg) Date: Sat, 12 Jan 2013 15:04:35 -0800 From: John-Mark Gurney To: David Xu Subject: Re: Fast sigblock (AKA rtld speedup) Message-ID: <20130112230435.GJ1410@funkthat.com> Mail-Followup-To: David Xu , Konstantin Belousov , arch@freebsd.org, toolchain@freebsd.org References: <20130107182235.GA65279@kib.kiev.ua> <50EBAA1F.9070303@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50EBAA1F.9070303@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sat, 12 Jan 2013 15:04:35 -0800 (PST) Cc: arch@freebsd.org, toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jan 2013 23:04:42 -0000 David Xu wrote this message on Tue, Jan 08, 2013 at 13:09 +0800: > and can not be freed until process is exited, the page is doubly > mapped into in kernel and userland, accessing the shared data > in kernel has zero overhead though. Don't forget that there are arches out there w/ VIVT caches which will probably eliminate most of the performance benifits if we have the same page mapped writable in two different virtual addresses.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."