From owner-freebsd-arch@freebsd.org Sun Feb 14 05:53:58 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 980D3AA0B97 for ; Sun, 14 Feb 2016 05:53:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 786C9E26 for ; Sun, 14 Feb 2016 05:53:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 76D67AA0B96; Sun, 14 Feb 2016 05:53:58 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C7C2AA0B94 for ; Sun, 14 Feb 2016 05:53:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x231.google.com (mail-qg0-x231.google.com [IPv6:2607:f8b0:400d:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21917E25 for ; Sun, 14 Feb 2016 05:53:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qg0-x231.google.com with SMTP id y89so91189685qge.2 for ; Sat, 13 Feb 2016 21:53:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=WzB4iIaVBqkYo1PLpvMolS6fzBy72oexlcWvF1RfQ9o=; b=c6PnAuyJsizlafsiQQ+YGo8VWW95WGAV9UY5gtkYLrQGgJovGPVAGhn+jbzAmBXd2a j8UhBY2h/jCci9M92v8rz7yBk63anXlxo4Vt9SCxXxMAX+s/0RRql+6vE1YRbdJ7gi3t bUNC+154muTxFiWKCIDIcyNf6TFppwrsm7wzkRxmZVzIwnuMidhRNMhrXJEqC6gWf2sP 1ASR3mwhft2CicTkPALk7IqFWTHBbCUjQLrDkANlUKFRna9BAVkFHn8eYPrL2mR5Va6W lBKdkdiEz8YJGYs8VtNtjyDaoyPQ4PMZyXemGtcjFM04jXWTJC75BdiMAsayQLoAmT3N Nwmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=WzB4iIaVBqkYo1PLpvMolS6fzBy72oexlcWvF1RfQ9o=; b=M9QOIzSGmxd+/zrjLZKO/Tt1YW+KhtUnpygcAT9EAXB0qDuD0tzkPAV2us32tz7QZU bmgcH4aQEM4RHSkogs3Mwa4XHlkVyF8OZqhGlmds/jRR1dTwFpD3kXnc8drfajj0dUt1 efiIzWSY6wNUgX4NF1xF7/QPRb8TatPI+SCDIO8In64UuK/HutPt2DjIMXBYxbt4sIqo 1bsyY86MgxvppFaNqL/mNi6/0C42/KWRAXNhY0AZXUwex/duFDz2NPDsO25kjzt11ox/ c2x358BZicM7ztttOKtUDECX/p1whq/DkNci+BQVdh3vPwLXEPyThAhBgeSyLNy6e/h6 UkPQ== X-Gm-Message-State: AG10YORvQIWS4DKuNtfijlJbmXf2XbKOEf697JaDOUVb2aJdZgptqqAKJLcRWhBvf8czDm6pxCzfAfQA+Awn0Q== MIME-Version: 1.0 X-Received: by 10.140.93.87 with SMTP id c81mr12737309qge.46.1455429236384; Sat, 13 Feb 2016 21:53:56 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.30.166 with HTTP; Sat, 13 Feb 2016 21:53:56 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: References: Date: Sat, 13 Feb 2016 22:53:56 -0700 X-Google-Sender-Auth: W0Ie7GlbXdL6f0eX89yye7x0UVE Message-ID: Subject: Re: OpenBSD mallocarray From: Warner Losh To: Ed Schouten Cc: C Turt , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Feb 2016 05:53:58 -0000 On Sat, Feb 13, 2016 at 6:21 AM, Ed Schouten wrote: > Hi there, > > 2016-02-01 20:57 GMT+01:00 C Turt : > > if ((nmemb >= MUL_NO_OVERFLOW || size >= MUL_NO_OVERFLOW) && > > nmemb > 0 && SIZE_MAX / nmemb < size) { > > In my opinion functions like these are good additions, as long as we > make sure that we stop importing garbage expressions like the one > above. It's already bad enough that we copy-pasted this gobbledygook > into fread(), fwrite(), calloc(), reallocarray(), etc. > > Both the latest versions of Clang and GCC support the following builtins: > > bool __builtin_add_overflow(type1 a, type2 b, type3 *res); > bool __builtin_sub_overflow(type1 a, type2 b, type3 *res); > bool __builtin_mul_overflow(type1 a, type2 b, type3 *res); > > These functions perform addition, subtraction and multiplication, > returning whether overflow has occurred in the process. This is a lot > more efficient (and readable) than the expression above, as it simply > uses the CPU's mul instruction, followed by a jump-on-overflow. > > GCC 4.2.1 doesn't support these builtins, but they can easily be > emulated by using static inline functions that use the code above. If > we want them to be type generic, we can use 's > __generic(), which either expands to C11's _Generic() or falls back to > GCC's __builtin_types_compatible_p()/__builtin_choose_expr(). > > I'd say it would make a lot of sense to add a new header file, e.g. > , that adds compiler-independent wrappers for these > builtins: > > #if recent version of GCC/Clang > #define add_overflow(a, b, res) __builtin_add_overflow(a, b, res) > #else > #define add_overflow(a, b, res) (__generic(*(res), unsigned int, ..., > ...)(a, b, res)) > #endif > This actually makes a great deal of sense to me. Warner From owner-freebsd-arch@freebsd.org Sun Feb 14 06:03:29 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF2FBAA0FCF for ; Sun, 14 Feb 2016 06:03:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CD5FB115D for ; Sun, 14 Feb 2016 06:03:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.ysv.freebsd.org (Postfix) id C9AD9AA0FCE; Sun, 14 Feb 2016 06:03:29 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C92C1AA0FCD for ; Sun, 14 Feb 2016 06:03:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x22c.google.com (mail-qg0-x22c.google.com [IPv6:2607:f8b0:400d:c04::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8299E115C for ; Sun, 14 Feb 2016 06:03:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qg0-x22c.google.com with SMTP id y9so91829058qgd.3 for ; Sat, 13 Feb 2016 22:03:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=f/IumtJLZyYoXoJqSHMZ30Qen4fsPyJO8+wWYluqLek=; b=acptvYK2QZ74h2LSfHHfL9C3a0on067/uKnZBBB6Ve7CuCZbzO38TG1qiRoGwqwI62 cxrslw77la+d0QmlQVG/mvd1VjNqKIVAoySRxGAk+KRk+OwXjcx8DeE5qS8caVDLXaoD NEpFThfjhgmu9pEpUZ4qd/qldNzeJ7ZGnwpJEnSCmYPi31RcCOU0loslU/2eJgqTKoTW sQ/yidTi3BoaqSKrtmTD71DQanyBM9uutKS2t+t7jsJCbD8qDg/a5aVDsf5TktaUGUh3 7MPQgtnkoNrVQsQTiaTir8b9HE/Wb5Bj1jlZ763dNfKCOIb5xPJGKmX4ap4kjRH/BM/6 lwhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=f/IumtJLZyYoXoJqSHMZ30Qen4fsPyJO8+wWYluqLek=; b=GTRrLXd+sf/RgvIP1AlR2l41lF5E6Lh6hygm5CF028lxXA/7Z3qM8Gm7YJ21uTMEb2 ObTcrsr6Tz9MFCUyN8hZePJUrGACwpZPoKJuu79kYJQaNpYVXOIU0Hac8Inkaw0UVHzn h7qP8xon1XucyBrfFYkrSyztOgP8vdtw1OLtv4yZmHktt8pYIWfGtTXAL5bFJtggCX4B J5QrCXYA/1bt3GJO9FWVGPTZI8GismkAhSXfRFgUgatOcy8JxvVE6kvyIOXvaI7umE7V /TrzFHJ29zscMQuwOD3RSnwfHNDQQjJRAHW6IMoQ0dojsn2C3NTbijJMwQ7QxhoqSG4t PUYw== X-Gm-Message-State: AG10YOSRGeAPP2wvmGEWlNDP/3WsKJwpnhAzJkCy5tpr/VIiBrfECLyJeTvqMNwxEIoqMTUNcEKZEdX55dPhDQ== MIME-Version: 1.0 X-Received: by 10.140.221.17 with SMTP id r17mr13335297qhb.94.1455429808595; Sat, 13 Feb 2016 22:03:28 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.30.166 with HTTP; Sat, 13 Feb 2016 22:03:28 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: References: Date: Sat, 13 Feb 2016 23:03:28 -0700 X-Google-Sender-Auth: VDdGeOwLSnYj0i6-9fLQZsHALr8 Message-ID: Subject: Re: OpenBSD mallocarray From: Warner Losh To: C Turt Cc: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Feb 2016 06:03:30 -0000 On Sat, Feb 13, 2016 at 2:18 AM, C Turt wrote: > "Except that you=E2=80=99d need to change all the code that was imported = into > the kernel to use mallocarray. The code Sony imported didn=E2=80=99t have= it > to start with." > > Although most of the PS4 dynamic linker is taken from userland FreeBSD > rtld-elf, the particular system call which contains the overflow, > sys_dynlib_prepare_dlclose, is a Sony extension which was not imported an= d > so would have been written from scratch with the intention of being kerne= l > code. Hence, I continue to believe that if it was common practice to use > mallocarray in the kernel, it would likely have been used here. > > "These are all good finds. And they are all mitigable with the > MALLOC_OVERFLOW > macro that was suggested earlier in this thread. Given the unique demands > of the > kernel, why should that not be the preferred method of dealing with this > stuff?" > > You are absolutely right that macros like MALLOC_OVERFLOW should be the > preferred way of catching integer overflows, and dealing with them > appropriately according to the unique context where the overflow occurs. = My > intention isn't to remove checks and rely on mallocarray to deal with the= m, > it is to have both, such that only if the initial checks are faulty they > will be caught in mallocarray, where the system should then intentionally > chose to panic rather than continue, leading to a probable heap overflow. > We're already seeing places in the kernel where extra cycles being uselessly consumed are interfering with performance needed to get to 100Gbps off disks and out the network port. While these code paths are unlikely to have a malloc in them (since they've been optimized in the push for 10Gbps and 40Gbps), extra, redundant checks, mindlessly applied throughout the kernel likely aren't the right answer either. > The problem I have is that certain parts of FreeBSD kernel flat out don't > live up to FreeBSD's reputation of having clean code written with securit= y > in mind. Look at `huft_build` from `sys/kern/inflate.c`, this has to be t= he > ugliest function in the whole of FreeBSD kernel, and there is an allocati= on > with the described pattern: > > if ((q =3D (struct huft *) malloc((z + 1) * sizeof(struct huft), M_GZIP, > M_WAITOK)) =3D=3D > (struct huft *) NULL) { > > What's more, this code has a history of containing vulnerabilities ( > https://www.freebsd.org/security/advisories/FreeBSD-SA-06:21.gzip.asc and > CVE-2009-2624). > > `z + 1` should be, and probably is, guaranteed by this code to be within > appropriate bounds. However, my proposal is that pieces of code like this > should be replaced with `mallocarray` so that there is absolutely no way > that this can ever overflow from the multiplication at least. > I still maintain that a check before this, or as Ed Shouten has pointed out, using an API that can do the multiplication and report overflow using the CPU facilities to do so would be a much better API, we'd address the performance issues I'm worried about. > Although there are many other ways that an allocation like this could > potentially be vulnerable: overflowing from the `+ 1`, or the count being > raced attacked if it wasn't a stack variable, for example, I believe that > using `mallocarray` would be an excellent start in pro-actively increasin= g > the security and code quality of the FreeBSD kernel. > I've still not seen an argument why this method is superior to the ones outlined by Bruce Evans or Ed Shouten. You have to change the code anyway, it would make more sense to make sure that your changes reflect what you are doing, rather than add an obfuscating function to the mix with a poor API for the kernel. Warner > On Thu, Feb 11, 2016 at 9:36 PM, Warner Losh wrote: > >> >> > On Feb 11, 2016, at 2:21 PM, C Turt wrote: >> > >> > I just wanted to post some real world examples of bugs which could be >> > mitigated with `mallocarray` to attract more interest. >> >> Let=E2=80=99s play devil=E2=80=99s advocate: since you have to make code= changes, how >> would mallocarray() be superior to the various MALLOC_OVERFLOW >> macro suggestions from earlier in the thread? Given that kernel code is >> somewhat different in what it needs to support? >> >> > My most meritable example of a real world attack from this behaviour >> would >> > be the sys_dynlib_prepare_dlclose kernel exploit for PS4 (PS4 OS is >> based >> > on FreeBSD 9.0). You may read my write-up of this exploit here: >> > http://cturt.github.io/dlclose-overflow.html >> > >> > The significance of this example is that if a `mallocarray` wrapper wa= s >> > available, and used here, the bug would not have been exploitable, >> because >> > it would have intentionally panicked instead of continuing with an und= er >> > allocated buffer. >> >> Except that you=E2=80=99d need to change all the code that was imported = into >> the kernel to use mallocarray. The code Sony imported didn=E2=80=99t hav= e it >> to start with. >> >> > You may think that this is clearly Sony's fault for not checking the >> count >> > at all, and that FreeBSD kernel code would never have a bug like this, >> > which is why I would like to mention that similar overflows can be >> possible >> > even if there initially appear to be sufficient bound checks performed= . >> > >> > There are several pieces of vulnerable code present in FreeBSD kernel >> > (albeit most of them are triggerable only as root so are not critical)= , >> > however I will use the file `/sys/kern/kern_alq.c` to demonstrate. The= re >> > are some checks on counts and sizes, but they are insufficient: >> > >> > [CODE]int >> > alq_open(struct alq **alqp, const char *file, struct ucred *cred, int >> cmode, >> > int size, int count) >> > { >> > int ret; >> > >> > KASSERT((count >=3D 0), ("%s: count < 0", __func__)); >> > >> > if (count > 0) { >> > if ((ret =3D alq_open_flags(alqp, file, cred, cmode, >> > size*count, 0)) =3D=3D 0) { >> > (*alqp)->aq_flags |=3D AQ_LEGACY; >> > (*alqp)->aq_entmax =3D count; >> > (*alqp)->aq_entlen =3D size; >> > } >> > >> > ... >> > >> > int >> > alq_open_flags(struct alq **alqp, const char *file, struct ucred *cred= , >> int >> > cmode, >> > int size, int flags) >> > { >> > ... >> > KASSERT((size > 0), ("%s: size <=3D 0", __func__)); >> > ... >> > alq->aq_buflen =3D size; >> > ... >> > alq->aq_entbuf =3D malloc(alq->aq_buflen, M_ALD, >> M_WAITOK|M_ZERO);[/CODE] >> > >> > The check on `count` being greater than or equal to 0 in `alq_open`, a= nd >> > the check for `size` being greater than 0 in `alq_open_flags` are cute= , >> but >> > they don't check for an overflow of `size*count`. >> > >> > This code path is reachable in several places, such as >> > `sysctl_debug_ktr_alq_enable`: >> > >> > [CODE]static int >> > sysctl_debug_ktr_alq_enable(SYSCTL_HANDLER_ARGS) >> > { >> > ... >> > error =3D alq_open(&ktr_alq, (const char *)ktr_alq_file, >> > req->td->td_ucred, ALQ_DEFAULT_CMODE, >> > sizeof(struct ktr_entry), ktr_alq_depth); >> > [/CODE] >> > >> > With `ktr_alq_depth` being controllable from userland (but only as >> root): >> > >> > [CODE]SYSCTL_INT(_debug_ktr, OID_AUTO, alq_depth, CTLFLAG_RW, >> > &ktr_alq_depth, 0, "Number of items in the write buffer");[/CODE] >> > >> > `sizeof(struct ktr_entry)` is 88 bytes. So theoretically if >> `ktr_alq_depth` >> > is set to `48806447`, we'll get an allocation of `0x100000028`, which >> > truncates to 0x28 =3D 40 bytes. A heap overflow would then possible wh= en >> this >> > buffer is iterated over with `aq_entmax` and `aq_entlen`. >> >> These are all good finds. And they are all mitigable with the >> MALLOC_OVERFLOW >> macro that was suggested earlier in this thread. Given the unique demand= s >> of the >> kernel, why should that not be the preferred method of dealing with this >> stuff? >> >> Warner >> >> >> > On Mon, Feb 1, 2016 at 7:57 PM, C Turt wrote: >> > >> >> I've recently started browsing the OpenBSD kernel source code, and ha= ve >> >> found the mallocarray function positively wonderful. I would like to >> >> discuss the possibility of getting this into FreeBSD kernel. >> >> >> >> For example, many parts of kernel code in FreeBSD use something like >> >> malloc(xxx * sizeof(struct xxx)). If xxx is 64bit and controllable by >> user, >> >> this allocation can easily overflow, resulting in a heap overflow >> later on. >> >> >> >> The mallocarray is a wrapper for malloc which can be used in this >> >> situations to detect an integer overflow before allocating: >> >> >> >> /* >> >> * Copyright (c) 2008 Otto Moerbeek >> >> * >> >> * Permission to use, copy, modify, and distribute this software for a= ny >> >> * purpose with or without fee is hereby granted, provided that the >> above >> >> * copyright notice and this permission notice appear in all copies. >> >> * >> >> * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL >> WARRANTIES >> >> * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF >> >> * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE >> FOR >> >> * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY >> DAMAGES >> >> * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN = AN >> >> * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OU= T >> OF >> >> * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. >> >> */ >> >> >> >> /* >> >> * This is sqrt(SIZE_MAX+1), as s1*s2 <=3D SIZE_MAX >> >> * if both s1 < MUL_NO_OVERFLOW and s2 < MUL_NO_OVERFLOW >> >> */ >> >> #define MUL_NO_OVERFLOW (1UL << (sizeof(size_t) * 4)) >> >> >> >> void * >> >> mallocarray(size_t nmemb, size_t size, int type, int flags) >> >> { >> >> if ((nmemb >=3D MUL_NO_OVERFLOW || size >=3D MUL_NO_OVERFLOW) && >> >> nmemb > 0 && SIZE_MAX / nmemb < size) { >> >> if (flags & M_CANFAIL) >> >> return (NULL); >> >> panic("mallocarray: overflow %zu * %zu", nmemb, size); >> >> } >> >> return (malloc(size * nmemb, type, flags)); >> >> } >> >> >> >> >> > _______________________________________________ >> > freebsd-arch@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-arch >> > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org= " >> >> > From owner-freebsd-arch@freebsd.org Sun Feb 14 12:56:29 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 602ADAA0496 for ; Sun, 14 Feb 2016 12:56:29 +0000 (UTC) (envelope-from avi.cohen@huawei.com) Received: from lhrrgout.huawei.com (lhrrgout.huawei.com [194.213.3.17]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "myname.my.domain", Issuer "www.mirapoint.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CF53210C2 for ; Sun, 14 Feb 2016 12:56:28 +0000 (UTC) (envelope-from avi.cohen@huawei.com) Received: from 172.18.7.190 (EHLO lhreml405-hub.china.huawei.com) ([172.18.7.190]) by lhrrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id CIN47641; Sun, 14 Feb 2016 12:55:17 +0000 (GMT) Received: from LHREML701-CAH.china.huawei.com (10.201.5.93) by lhreml405-hub.china.huawei.com (10.201.5.242) with Microsoft SMTP Server (TLS) id 14.3.235.1; Sun, 14 Feb 2016 12:55:17 +0000 Received: from LHREML501-MBB.china.huawei.com ([10.125.30.98]) by lhreml701-cah.china.huawei.com ([10.201.5.93]) with mapi id 14.03.0235.001; Sun, 14 Feb 2016 12:55:12 +0000 From: Avi Cohen To: "freebsd-arch@freebsd.org" Subject: Paravirtualized KVM clock Thread-Topic: Paravirtualized KVM clock Thread-Index: AdFnJu/nPBQ/VGujSleckJL4Tt9PSQ== Date: Sun, 14 Feb 2016 12:55:11 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.200.201.159] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.56C07936.003D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: bd8bbf2455fda65cd4acb078e7c2b67d Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Feb 2016 12:56:29 -0000 Hello My name is Avi Cohen and I work in toganetworks which is part of Huawei ERC= . 1. My VM is Fedora 21 and the host is Fedora 22 (kernels are 4.1 and= 4.2 accordingly) 2. My Intel core processor supports the constant-tsc. 3. The default clock in the guest is kvm-clock . When I run a test to measure packet delay between the host and the VM (wh= ich normally should be about 5 to 10 micro-sec) - I see about 800 mili-s= ec delay I see that delta time between the VM and host is about 800ms - which means= that the sync. Accuracy is bad What accuracy do I have to expect with this clocking mode (kvm-clock) ? te= ns of ms/us/ns? Maybe I've missed some configuration to achieve a better accuracy ? Thanks and Regards, Avi From owner-freebsd-arch@freebsd.org Mon Feb 15 14:17:37 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB473AA995E for ; Mon, 15 Feb 2016 14:17:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CB8111C1B for ; Mon, 15 Feb 2016 14:17:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: by mailman.ysv.freebsd.org (Postfix) id C6992AA995C; Mon, 15 Feb 2016 14:17:37 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6153AA995B; Mon, 15 Feb 2016 14:17:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 10E551C16; Mon, 15 Feb 2016 14:17:30 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.9/8.14.9) with ESMTP id u1FEHKDT031345; Mon, 15 Feb 2016 14:17:20 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id u1FEHKih003396; Mon, 15 Feb 2016 14:17:20 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id u1FEHKwL003392; Mon, 15 Feb 2016 14:17:20 GMT Date: Mon, 15 Feb 2016 14:17:20 GMT Message-Id: <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> From: Martin Simmons To: Konstantin Belousov CC: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org In-reply-to: <20160213143815.GB91220@kib.kiev.ua> (message from Konstantin Belousov on Sat, 13 Feb 2016 16:38:15 +0200) Subject: Re: libthr shared locks References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 14:17:38 -0000 Is pthread_barrier_destroy making the wrong comparison? + if (barrier == THR_PSHARED_PTR) { I think this should be *barrier. Also, a general question: why not use some flag in the barrier (and other objects) to indicate pshared, removing the need for __thr_pshared_offpage except in init? __Martin From owner-freebsd-arch@freebsd.org Mon Feb 15 14:44:17 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6A79AA89E7 for ; Mon, 15 Feb 2016 14:44:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C2AA512D4 for ; Mon, 15 Feb 2016 14:44:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id BE2FCAA89E5; Mon, 15 Feb 2016 14:44:17 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDB16AA89E4; Mon, 15 Feb 2016 14:44:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59EAC12D3; Mon, 15 Feb 2016 14:44:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1FEiAQQ029016 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 15 Feb 2016 16:44:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1FEiAQQ029016 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1FEiAfr029015; Mon, 15 Feb 2016 16:44:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 15 Feb 2016 16:44:10 +0200 From: Konstantin Belousov To: Martin Simmons Cc: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org Subject: Re: libthr shared locks Message-ID: <20160215144410.GT91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 14:44:17 -0000 On Mon, Feb 15, 2016 at 02:17:20PM +0000, Martin Simmons wrote: > Is pthread_barrier_destroy making the wrong comparison? > > + if (barrier == THR_PSHARED_PTR) { > > I think this should be *barrier. You are right, thank you for noticing. I uploaded https://www.kib.kiev.ua/kib/pshared/pshared.3.patch > > Also, a general question: why not use some flag in the barrier (and other > objects) to indicate pshared, removing the need for __thr_pshared_offpage > except in init? But where would I keep the object ? All that I have with the current ABI is a single pointer, which de facto behaves like the flag which you proposed. It is either real pointer or (if set to some specific value impossible for a valid pointer) there is an offpage. From owner-freebsd-arch@freebsd.org Mon Feb 15 17:35:38 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C856AA9EA1 for ; Mon, 15 Feb 2016 17:35:38 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7A97EA68 for ; Mon, 15 Feb 2016 17:35:38 +0000 (UTC) (envelope-from martin@lispworks.com) Received: by mailman.ysv.freebsd.org (Postfix) id 779E1AA9E9E; Mon, 15 Feb 2016 17:35:38 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77284AA9E9C; Mon, 15 Feb 2016 17:35:38 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 1E261A66; Mon, 15 Feb 2016 17:35:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.9/8.14.9) with ESMTP id u1FHZXVA039524; Mon, 15 Feb 2016 17:35:33 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id u1FHZXwV006194; Mon, 15 Feb 2016 17:35:33 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id u1FHZXKV006190; Mon, 15 Feb 2016 17:35:33 GMT Date: Mon, 15 Feb 2016 17:35:33 GMT Message-Id: <201602151735.u1FHZXKV006190@higson.cam.lispworks.com> From: Martin Simmons To: Konstantin Belousov CC: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org In-reply-to: <20160215144410.GT91220@kib.kiev.ua> (message from Konstantin Belousov on Mon, 15 Feb 2016 16:44:10 +0200) Subject: Re: libthr shared locks References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> <20160215144410.GT91220@kib.kiev.ua> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 17:35:38 -0000 >>>>> On Mon, 15 Feb 2016 16:44:10 +0200, Konstantin Belousov said: > > On Mon, Feb 15, 2016 at 02:17:20PM +0000, Martin Simmons wrote: > > > > Also, a general question: why not use some flag in the barrier (and other > > objects) to indicate pshared, removing the need for __thr_pshared_offpage > > except in init? > > But where would I keep the object ? All that I have with the current > ABI is a single pointer, which de facto behaves like the flag which you > proposed. It is either real pointer or (if set to some specific value > impossible for a valid pointer) there is an offpage. I'm probably missing something, but I was thinking pthread_barrier_init would do something like if ( attr is PTHREAD_PROCESS_PRIVATE ) { bar = calloc(1, sizeof(struct pthread_barrier)); pshared = 0; } else { bar = __thr_pshared_offpage(barrier, 1); pshared = 1; } bar->psharedflag = pshared; *barrier = bar; Then pthread_barrier_destroy would use the psharedflag slot to decide how to free it and pthread_barrier_wait would need no changes. From owner-freebsd-arch@freebsd.org Mon Feb 15 17:56:28 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6F56AA8BAD for ; Mon, 15 Feb 2016 17:56:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 909891783 for ; Mon, 15 Feb 2016 17:56:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 8B968AA8BAA; Mon, 15 Feb 2016 17:56:28 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B0FDAA8BA8; Mon, 15 Feb 2016 17:56:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0ACB3177E; Mon, 15 Feb 2016 17:56:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1FHuMk4081577 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 15 Feb 2016 19:56:22 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1FHuMk4081577 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1FHuL2h081576; Mon, 15 Feb 2016 19:56:21 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 15 Feb 2016 19:56:21 +0200 From: Konstantin Belousov To: Martin Simmons Cc: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org Subject: Re: libthr shared locks Message-ID: <20160215175621.GU91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> <20160215144410.GT91220@kib.kiev.ua> <201602151735.u1FHZXKV006190@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201602151735.u1FHZXKV006190@higson.cam.lispworks.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 17:56:28 -0000 On Mon, Feb 15, 2016 at 05:35:33PM +0000, Martin Simmons wrote: > >>>>> On Mon, 15 Feb 2016 16:44:10 +0200, Konstantin Belousov said: > > > > On Mon, Feb 15, 2016 at 02:17:20PM +0000, Martin Simmons wrote: > > > > > > Also, a general question: why not use some flag in the barrier (and other > > > objects) to indicate pshared, removing the need for __thr_pshared_offpage > > > except in init? > > > > But where would I keep the object ? All that I have with the current > > ABI is a single pointer, which de facto behaves like the flag which you > > proposed. It is either real pointer or (if set to some specific value > > impossible for a valid pointer) there is an offpage. > > I'm probably missing something, but I was thinking pthread_barrier_init would > do something like > > if ( attr is PTHREAD_PROCESS_PRIVATE ) { > bar = calloc(1, sizeof(struct pthread_barrier)); > pshared = 0; > } else { > bar = __thr_pshared_offpage(barrier, 1); > pshared = 1; > } > bar->psharedflag = pshared; > *barrier = bar; > > Then pthread_barrier_destroy would use the psharedflag slot to decide how to > free it and pthread_barrier_wait would need no changes. A process which has the page where the initialized pthread_barrier_t is located, must be able to operate on the barrier. Now, look at your scheme. One process which executed pthread_barrier_init(), performed what you proposed. What should do the pthread_barrier_wait() call in another process, which shares the 'barrier' with the first process, but does not share the whole address space ? After your pthread_barrier_init() executed, barrier contains the address of the object (off-page) in the other address space, for that process. From owner-freebsd-arch@freebsd.org Mon Feb 15 21:39:27 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02852AA8652 for ; Mon, 15 Feb 2016 21:39:27 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E835533B for ; Mon, 15 Feb 2016 21:39:26 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id E50E4AA8651; Mon, 15 Feb 2016 21:39:26 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E48F8AA864E; Mon, 15 Feb 2016 21:39:26 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id D025F33A; Mon, 15 Feb 2016 21:39:26 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from ford.home.vangyzen.net (unknown [76.164.15.242]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 8D38F56B12; Mon, 15 Feb 2016 15:39:19 -0600 (CST) Subject: Re: libthr shared locks To: Konstantin Belousov , threads@freebsd.org, arch@freebsd.org References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> From: Eric van Gyzen X-Enigmail-Draft-Status: N1110 Message-ID: <56C24586.9050906@FreeBSD.org> Date: Mon, 15 Feb 2016 15:39:18 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56BE69B8.9020808@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 21:39:27 -0000 On 02/12/16 05:24 PM, Eric van Gyzen wrote: > On 12/23/2015 11:25, Konstantin Belousov wrote: >> The implementation in the patch >> https://www.kib.kiev.ua/kib/pshared/pshared.1.patch >> gives shared mutexes, condvars, rwlocks and barriers. > I reviewed everything except kern_umtx.c, which I plan to review on > Monday. My only comment on kern_umtx.c is, why are the permission checks compiled out? I also reviewed versions 2 and 3; the revisions look fine. Thanks for the chance to review, and for waiting for me to find time. Eric From owner-freebsd-arch@freebsd.org Tue Feb 16 11:32:33 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32B8BAAAAA7 for ; Tue, 16 Feb 2016 11:32:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1D89E1EE6 for ; Tue, 16 Feb 2016 11:32:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1D34CAAAAA5; Tue, 16 Feb 2016 11:32:33 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CBF3AAAAA2; Tue, 16 Feb 2016 11:32:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 918D01EE4; Tue, 16 Feb 2016 11:32:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1GBWMsR011473 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 16 Feb 2016 13:32:23 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1GBWMsR011473 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1GBWMnp011470; Tue, 16 Feb 2016 13:32:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 16 Feb 2016 13:32:22 +0200 From: Konstantin Belousov To: Eric van Gyzen Cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20160216113222.GY91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56C24586.9050906@FreeBSD.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 11:32:33 -0000 On Mon, Feb 15, 2016 at 03:39:18PM -0600, Eric van Gyzen wrote: > My only comment on kern_umtx.c is, why are the permission checks compiled out? Assume that we changed the ABI of libthr and shared locks do not require an offpage. Then, access to the locks is completely controlled by the access to the page containing the lock. If a process can mmap the page, it can own the lock. >From this point of view, access to the offpage shared memory object is the same as the access to the key page. Which is the reason to not include the access permissions checks. On the other hand, if you have something in kernel, which also owns a reference on ucred (for other means), you sure consider whether the usual access control is appropriate. Mostly, I leave the #ifdef-ed checks to reconsider it later, which worked since you asked the question. I definitely do not see much use of the shm_access() checks, but I am not completely sure about possible mac policies utilization there, although I do not really think they could be usefully attached to the app-level locks. > I also reviewed versions 2 and 3; the revisions look fine. Thank you very much. From owner-freebsd-arch@freebsd.org Tue Feb 16 14:56:44 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2541DAAA805 for ; Tue, 16 Feb 2016 14:56:44 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 15BD383 for ; Tue, 16 Feb 2016 14:56:44 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 132C4AAA802; Tue, 16 Feb 2016 14:56:44 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 129D2AAA801; Tue, 16 Feb 2016 14:56:44 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id F155A82; Tue, 16 Feb 2016 14:56:43 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 2A87556B13; Tue, 16 Feb 2016 08:56:37 -0600 (CST) Subject: Re: libthr shared locks To: Konstantin Belousov References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> Cc: threads@freebsd.org, arch@freebsd.org From: Eric van Gyzen X-Enigmail-Draft-Status: N1110 Message-ID: <56C338A0.2060205@FreeBSD.org> Date: Tue, 16 Feb 2016 08:56:32 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <20160216113222.GY91220@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 14:56:44 -0000 On 02/16/2016 05:32, Konstantin Belousov wrote: > On Mon, Feb 15, 2016 at 03:39:18PM -0600, Eric van Gyzen wrote: >> My only comment on kern_umtx.c is, why are the permission checks compiled out? > Assume that we changed the ABI of libthr and shared locks do not require > an offpage. Then, access to the locks is completely controlled by the > access to the page containing the lock. If a process can mmap the page, > it can own the lock. > > From this point of view, access to the offpage shared memory object > is the same as the access to the key page. Which is the reason to not > include the access permissions checks. This makes sense. > On the other hand, if you have something in kernel, which also owns a > reference on ucred (for other means), you sure consider whether the usual > access control is appropriate. This sounds wise. I'll keep it in mind. > I > definitely do not see much use of the shm_access() checks, but I am not > completely sure about possible mac policies utilization there, although > I do not really think they could be usefully attached to the app-level > locks. I would tend to agree, but I haven't used MAC for several years, so I'm not sure either. Cheers, Eric From owner-freebsd-arch@freebsd.org Tue Feb 16 16:17:57 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20DF6AA9C0C for ; Tue, 16 Feb 2016 16:17:57 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0E9E8FD3 for ; Tue, 16 Feb 2016 16:17:57 +0000 (UTC) (envelope-from martin@lispworks.com) Received: by mailman.ysv.freebsd.org (Postfix) id 01C19AA9C0A; Tue, 16 Feb 2016 16:17:57 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 013CFAA9C09; Tue, 16 Feb 2016 16:17:57 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 9BB3AFD1; Tue, 16 Feb 2016 16:17:55 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.9/8.14.9) with ESMTP id u1GGHkx5039283; Tue, 16 Feb 2016 16:17:46 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id u1GGHkPF023638; Tue, 16 Feb 2016 16:17:46 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id u1GGHkil023634; Tue, 16 Feb 2016 16:17:46 GMT Date: Tue, 16 Feb 2016 16:17:46 GMT Message-Id: <201602161617.u1GGHkil023634@higson.cam.lispworks.com> From: Martin Simmons To: Konstantin Belousov CC: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org In-reply-to: <20160215175621.GU91220@kib.kiev.ua> (message from Konstantin Belousov on Mon, 15 Feb 2016 19:56:21 +0200) Subject: Re: libthr shared locks References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> <20160215144410.GT91220@kib.kiev.ua> <201602151735.u1FHZXKV006190@higson.cam.lispworks.com> <20160215175621.GU91220@kib.kiev.ua> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 16:17:57 -0000 >>>>> On Mon, 15 Feb 2016 19:56:21 +0200, Konstantin Belousov said: > > One process which executed pthread_barrier_init(), performed what you > proposed. What should do the pthread_barrier_wait() call in another > process, which shares the 'barrier' with the first process, but does > not share the whole address space ? After your pthread_barrier_init() > executed, barrier contains the address of the object (off-page) in the > other address space, for that process. Ah, sorry, I understand now (the init functions are called before any sharing). How should the destroy functions be used by the processes? I.e. should only the "last" process call destroy or can every process call it? From owner-freebsd-arch@freebsd.org Tue Feb 16 17:20:36 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B416AAAA51E for ; Tue, 16 Feb 2016 17:20:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 969C2159A for ; Tue, 16 Feb 2016 17:20:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 86BFFB9BA; Tue, 16 Feb 2016 12:20:35 -0500 (EST) From: John Baldwin To: Jilles Tjoelker Cc: freebsd-arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Date: Mon, 15 Feb 2016 17:10:19 -0800 Message-ID: <18990810.Gs6uxxBMa9@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160211223238.GA73182@stack.nl> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <2604669.yblYTI1T32@ralph.baldwin.cx> <20160211223238.GA73182@stack.nl> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 16 Feb 2016 12:20:35 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 17:20:36 -0000 On Thursday, February 11, 2016 11:32:38 PM Jilles Tjoelker wrote: > On Wed, Feb 10, 2016 at 04:21:30PM -0800, John Baldwin wrote: > > I've pushed a new version with a vfs.aio.enable_unsafe sysctl > > (defaults to off). If you think this is ok then I'll work on cleaning > > up some of the module bits (removing the unload support mostly). I > > figure marking the syscalls as STD and removing all the helper stuff > > can be done as a followup commit to reduce extra noise in this diff > > (which is large enough on its own). > > > https://github.com/freebsd/freebsd/compare/master...bsdjhb:aio_rework > > > (Also, for the inital commit I will leave out the socket protocol > > hook. That can be added later.) > > Looks good to me, except that the error should be [ENOTSUP] instead of > the overloaded [EINVAL]. Ok. I've done some further cleanups and posted a commit candidate at https://reviews.freebsd.org/D5289 for anyone else who is interested. -- John Baldwin From owner-freebsd-arch@freebsd.org Tue Feb 16 17:27:19 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E379DAAA7BE for ; Tue, 16 Feb 2016 17:27:19 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D151A1D36 for ; Tue, 16 Feb 2016 17:27:19 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CC3C6AAA7BB; Tue, 16 Feb 2016 17:27:19 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBB53AAA7BA; Tue, 16 Feb 2016 17:27:19 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 960F71D33; Tue, 16 Feb 2016 17:27:19 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id u1GHRC30058087; Tue, 16 Feb 2016 12:27:12 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Tue, 16 Feb 2016 12:27:12 -0500 (EST) Date: Tue, 16 Feb 2016 12:27:12 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20160216113222.GY91220@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 17:27:20 -0000 On Tue, 16 Feb 2016, Konstantin Belousov wrote: > On Mon, Feb 15, 2016 at 03:39:18PM -0600, Eric van Gyzen wrote: >> My only comment on kern_umtx.c is, why are the permission checks compiled out? > > Assume that we changed the ABI of libthr and shared locks do not require > an offpage. How are you coming with that? I thought maybe you were going to think about making the synch types structs, but not actually changing the implementation (yet). -- DE From owner-freebsd-arch@freebsd.org Tue Feb 16 20:50:28 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25DCEAAB28F for ; Tue, 16 Feb 2016 20:50:28 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0BA8D18D6 for ; Tue, 16 Feb 2016 20:50:28 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 07DE4AAB28E; Tue, 16 Feb 2016 20:50:28 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1D02AAB28D for ; Tue, 16 Feb 2016 20:50:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD45E18D5 for ; Tue, 16 Feb 2016 20:50:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 134F0B93E for ; Tue, 16 Feb 2016 15:50:26 -0500 (EST) From: John Baldwin To: arch@freebsd.org Subject: Starting APs earlier during boot Date: Tue, 16 Feb 2016 12:50:22 -0800 Message-ID: <1730061.8Ii36ORVKt@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 16 Feb 2016 15:50:26 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2016 20:50:28 -0000 Currently the kernel bootstraps the non-boot processors fairly early in the SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We currently release the APs as one of the last steps at SI_SUB_SMP. On the one hand this removes much of the need for synchronization while SYSINITs are running since SYSINITs basically assume they are single-threaded. However, it also enforces some odd quirks. Several places that deal with per-CPU resources have to split initialization up so that the BSP init happens in one SYSINIT and the initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. Another issue that is becoming more prominent on x86 (and probably will also affect other platforms if it isn't already) is that to support working interrupts for interrupt config hooks we bind all interrupts to the BSP during boot and only distribute them among other CPUs near the end at SI_SUB_SMP. This is especially problematic with drivers for modern hardware allocating num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug 190 IDT vectors available for device interrupts, so in theory we should be able to tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 interrupts for every CPU and we should still be fine). However, if you have, say, 32 cores in a system, then you can only handle about 5 drivers doing this before you run out of vectors on CPU 0. Longer term we would also like to eventually have most drivers attach in the same environment during boot as during post-boot. Right now post-boot is quite different as all CPUs are running, interrupts work, etc. One of the goals of multipass support for new-bus is to help us get there by probing enough hardware to get timers working and starting the scheduler before probing the rest of the devices. That goal isn't quite realized yet. However, we can run a slightly simpler version of our scheduler before timers are working. In fact, sleep/wakeup work just fine fairly early (we allocate the necessary structures at SI_SUB_KMEM which is before the APs are even started). Once idle threads are created and ready we could in theory let the APs startup and run other threads. You just don't have working timeouts. OTOH, you can sort of simulate timeouts if you modify the scheduler to yield the CPU instead of blocking the thread for a sleep with a timeout. The effect would be for threads that do sleeps with a timeout to fall back to polling before timers are working. In practice, all of the early kernel threads use sleeps without timeouts when idle so this doesn't really matter. I've implemented these changes and tested them for x86. For x86 at least AP startup needed some bits of the interrupt infrastructure in place, so I moved SI_SUB_SMP up to after SI_SUB_INTR but before SI_SUB_SOFTINTR. I modified the *sleep() and cv_*wait*() routines to not always bail if cold is true. Instead, sleeps without a timeout are permitted to sleep "normally". Sleeps with a timeout drop their interlock and yield the CPU (but remain runnable). Since APs are now fully running this means interrupts are now routed to all CPUs from the get go removing the need for the post-boot shuffle. This also resolves the issue of running out of IDT vectors on the boot CPU. I believe that adopting other platforms for this change should be relatively simple, but we should do that before committing the full patch. I do think that some parts of the patch (such as the changes to the sleep routines, and using SI_SUB_LAST instead of SI_SUB_SMP as a catch-all SYSINIT) can be committed now without breaking anything. However, I'd like feedback on the general idea and if it is acceptable I'd like to coordinate testing with other platforms so this can go into the tree. The current changes are in the 'ap_startup' branch at github/bsdjhb/freebsd. You can view them here: https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Feb 17 05:33:28 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 327C6AABF52 for ; Wed, 17 Feb 2016 05:33:28 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2142A1D5E for ; Wed, 17 Feb 2016 05:33:28 +0000 (UTC) (envelope-from julian@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 20889AABF4E; Wed, 17 Feb 2016 05:33:28 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 202F1AABF4D for ; Wed, 17 Feb 2016 05:33:28 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F2AD51D59; Wed, 17 Feb 2016 05:33:27 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from julian-mbp3.pixel8networks.com (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u1H5XKX0078982 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 16 Feb 2016 21:33:21 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Starting APs earlier during boot To: John Baldwin , arch@freebsd.org References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> From: Julian Elischer Message-ID: <56C4061B.6010601@freebsd.org> Date: Tue, 16 Feb 2016 21:33:15 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <1730061.8Ii36ORVKt@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 05:33:28 -0000 On 16/02/2016 12:50 PM, John Baldwin wrote: > Currently the kernel bootstraps the non-boot processors fairly early in the > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We currently > release the APs as one of the last steps at SI_SUB_SMP. On the one hand this > removes much of the need for synchronization while SYSINITs are running since > SYSINITs basically assume they are single-threaded. However, it also enforces > some odd quirks. Several places that deal with per-CPU resources have to > split initialization up so that the BSP init happens in one SYSINIT and the > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > Another issue that is becoming more prominent on x86 (and probably will also [...] what is the goal? cleaner code? faster boot? > From owner-freebsd-arch@freebsd.org Wed Feb 17 06:23:19 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76B17AAB3F7 for ; Wed, 17 Feb 2016 06:23:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5746B1F91 for ; Wed, 17 Feb 2016 06:23:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5657FAAB3F6; Wed, 17 Feb 2016 06:23:19 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C0C0AAB3F5 for ; Wed, 17 Feb 2016 06:23:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0159F1F8E for ; Wed, 17 Feb 2016 06:23:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qg0-x232.google.com with SMTP id y89so5125947qge.2 for ; Tue, 16 Feb 2016 22:23:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hDEGL8+ZUEFkHm7xkhtOGCovsU6km8mzYD04KAvoG/M=; b=c15GY4uOm12LaKR7AZ6bQ4OhzWvbkfC4zBVbYBtkZZGYWNMB/PadpR9VQGwex4KPq9 pzCqeBDw5xW08kN2rjmgfGTK3+EzbRoRlwh+pUhO3tD5JhqDfZERiOFkvoq3CHUuNOpN zO4vad2/i/n7o4KzSJwGIbP9DUcGmATZfngA6xRE6Sp2ULsnin/yPnyThM0/pmaZ24tv SuIKvyAxAsec4GCzdsPwPK8ndjok7SzDfKDCyj8vVyXj3pzywLaUXXjOBjzWzhm5VOoM C2zmLsMSdmoIhrtvuvqKqFmOEHJwoAgzwj9j0AGlK8MwA9ccb6Ke9Ylh7JeRiH73pPSk oXFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hDEGL8+ZUEFkHm7xkhtOGCovsU6km8mzYD04KAvoG/M=; b=cqhd0n5IDILu+hXwpLn+dhiqqD70WbNGrI3hioscPnJvmrZ/I+5+qy9fZojTPXhzs4 vtr6Whku1vwEpbn7wXRb4B9jxsZFVYEGgfvrkUXNooBxiufaEBsbX73GXy0gWYkOtEdE ZX6By75jlOUiItqrxr291Kh7b0lMq5amnjDTdNjrW0h7NRjwYEOmIpREBL1Qfg0fUTAK vu1pXkM5WDzOt2yrifZ5YfyNBEV3K3iY7cobZxzYjEfSnjHF7KoulNFHAuX6qEuAhlUE 3UgAxFIduuNviFvW+PgAweNgMIPuwy3qTefTiXkn/Hfg946i1MlXd9y8ab8hgaihUTuu LfJw== X-Gm-Message-State: AG10YOSsZzhLMQNawz7+HRhZ11GBDpB9rtg/QrVASlM2nmG2BVhBiiK5e3xoUH3+bovcII0VhwoxiDrTcmultg== MIME-Version: 1.0 X-Received: by 10.140.229.209 with SMTP id z200mr33965415qhb.40.1455690197834; Tue, 16 Feb 2016 22:23:17 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.30.166 with HTTP; Tue, 16 Feb 2016 22:23:17 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: <56C4061B.6010601@freebsd.org> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <56C4061B.6010601@freebsd.org> Date: Tue, 16 Feb 2016 23:23:17 -0700 X-Google-Sender-Auth: ZEu8yDeXmZR68Ti6iKgSZwcdUIw Message-ID: Subject: Re: Starting APs earlier during boot From: Warner Losh To: Julian Elischer Cc: John Baldwin , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 06:23:19 -0000 On Tue, Feb 16, 2016 at 10:33 PM, Julian Elischer wrote: > On 16/02/2016 12:50 PM, John Baldwin wrote: > >> Currently the kernel bootstraps the non-boot processors fairly early in >> the >> SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We >> currently >> release the APs as one of the last steps at SI_SUB_SMP. On the one hand >> this >> removes much of the need for synchronization while SYSINITs are running >> since >> SYSINITs basically assume they are single-threaded. However, it also >> enforces >> some odd quirks. Several places that deal with per-CPU resources have to >> split initialization up so that the BSP init happens in one SYSINIT and >> the >> initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. >> >> Another issue that is becoming more prominent on x86 (and probably will >> also >> > [...] > > what is the goal? cleaner code? faster boot? Two goals were in his original email. (1) Start APs earlier so we can avoid issues with interrupt allocation (we're currently hitting limits of 160 interrupts when only one is active). (2) Make allocations more regular between startup and later loading drivers later. Right now some drivers defer a lot of work so they can allocate things at a time when all the resources are available. This helps make that code more regular and actually the same between the different cases. It has little to do with a faster boot, though it might enable parallel newbus tree enumeration if that ever gets properly locked. Warner From owner-freebsd-arch@freebsd.org Wed Feb 17 09:42:51 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7EFC3AAB4EC for ; Wed, 17 Feb 2016 09:42:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6A14B1527 for ; Wed, 17 Feb 2016 09:42:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 69101AAB4EA; Wed, 17 Feb 2016 09:42:51 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 689B5AAB4E9 for ; Wed, 17 Feb 2016 09:42:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 124791525; Wed, 17 Feb 2016 09:42:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1H9gflt094594 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 17 Feb 2016 11:42:42 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1H9gflt094594 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1H9gfvN094593; Wed, 17 Feb 2016 11:42:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 17 Feb 2016 11:42:41 +0200 From: Konstantin Belousov To: John Baldwin Cc: arch@freebsd.org Subject: Re: Starting APs earlier during boot Message-ID: <20160217094241.GF91220@kib.kiev.ua> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1730061.8Ii36ORVKt@ralph.baldwin.cx> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 09:42:51 -0000 On Tue, Feb 16, 2016 at 12:50:22PM -0800, John Baldwin wrote: > Currently the kernel bootstraps the non-boot processors fairly early in the > SI_SUB_CPU SYSINIT. The APs then spin waiting to be "released". We currently > release the APs as one of the last steps at SI_SUB_SMP. On the one hand this > removes much of the need for synchronization while SYSINITs are running since > SYSINITs basically assume they are single-threaded. However, it also enforces > some odd quirks. Several places that deal with per-CPU resources have to > split initialization up so that the BSP init happens in one SYSINIT and the > initialization of the APs happens in a second SYSINIT at SI_SUB_SMP. > > Another issue that is becoming more prominent on x86 (and probably will also > affect other platforms if it isn't already) is that to support working > interrupts for interrupt config hooks we bind all interrupts to the BSP during > boot and only distribute them among other CPUs near the end at SI_SUB_SMP. > This is especially problematic with drivers for modern hardware allocating > num(CPUs) interrupts (hoping to use one per CPU). On x86 we have aboug 190 > IDT vectors available for device interrupts, so in theory we should be able to > tolerate a lot of drivers doing this (e.g. 60 drivers could allocate 3 > interrupts for every CPU and we should still be fine). However, if you have, > say, 32 cores in a system, then you can only handle about 5 drivers doing > this before you run out of vectors on CPU 0. > > Longer term we would also like to eventually have most drivers attach in the > same environment during boot as during post-boot. Right now post-boot is > quite different as all CPUs are running, interrupts work, etc. One of the > goals of multipass support for new-bus is to help us get there by probing > enough hardware to get timers working and starting the scheduler before > probing the rest of the devices. That goal isn't quite realized yet. > > However, we can run a slightly simpler version of our scheduler before > timers are working. In fact, sleep/wakeup work just fine fairly early (we > allocate the necessary structures at SI_SUB_KMEM which is before the APs > are even started). Once idle threads are created and ready we could in > theory let the APs startup and run other threads. You just don't have working > timeouts. OTOH, you can sort of simulate timeouts if you modify the scheduler > to yield the CPU instead of blocking the thread for a sleep with a timeout. > The effect would be for threads that do sleeps with a timeout to fall back to > polling before timers are working. In practice, all of the early kernel > threads use sleeps without timeouts when idle so this doesn't really matter. I understand that timeouts can be somewhat simulated this way. But I do not quite understand how generic scheduling can work without (timer) interrupts. Suppose that we have two threads 1 and 2 of the same priority, both runnable, and due to some event thread 2 preempted thread 1. If thread 2 just runs without calling the preempt functions like msleep, what would guarentee that thread 1 eventually gets it CPU slice ? E.g. there might be no interrupts set up yet, and idle thread on UP gets on CPU, then the whole boot process could deadlock. > > I've implemented these changes and tested them for x86. For x86 at least > AP startup needed some bits of the interrupt infrastructure in place, so > I moved SI_SUB_SMP up to after SI_SUB_INTR but before SI_SUB_SOFTINTR. I > modified the *sleep() and cv_*wait*() routines to not always bail if cold > is true. Instead, sleeps without a timeout are permitted to sleep > "normally". Sleeps with a timeout drop their interlock and yield the > CPU (but remain runnable). Since APs are now fully running this means > interrupts are now routed to all CPUs from the get go removing the need for > the post-boot shuffle. This also resolves the issue of running out of IDT > vectors on the boot CPU. > > I believe that adopting other platforms for this change should be relatively > simple, but we should do that before committing the full patch. I do think > that some parts of the patch (such as the changes to the sleep routines, and > using SI_SUB_LAST instead of SI_SUB_SMP as a catch-all SYSINIT) can be > committed now without breaking anything. > > However, I'd like feedback on the general idea and if it is acceptable I'd > like to coordinate testing with other platforms so this can go into the > tree. > > The current changes are in the 'ap_startup' branch at github/bsdjhb/freebsd. > You can view them here: > > https://github.com/bsdjhb/freebsd/compare/master...bsdjhb:ap_startup > > -- > John Baldwin > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@freebsd.org Wed Feb 17 09:45:52 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC5A2AAB61F for ; Wed, 17 Feb 2016 09:45:52 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CC4F21639 for ; Wed, 17 Feb 2016 09:45:52 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: by mailman.ysv.freebsd.org (Postfix) id C9608AAB61E; Wed, 17 Feb 2016 09:45:52 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8F98AAB61D for ; Wed, 17 Feb 2016 09:45:52 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 949DB1638; Wed, 17 Feb 2016 09:45:52 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id F38D04F865; Wed, 17 Feb 2016 09:45:44 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTP id u1H9jhVh041150; Wed, 17 Feb 2016 09:45:43 GMT (envelope-from phk@phk.freebsd.dk) To: Konstantin Belousov cc: John Baldwin , arch@freebsd.org Subject: Re: Starting APs earlier during boot In-reply-to: <20160217094241.GF91220@kib.kiev.ua> From: "Poul-Henning Kamp" References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <20160217094241.GF91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <41148.1455702343.1@critter.freebsd.dk> Date: Wed, 17 Feb 2016 09:45:43 +0000 Message-ID: <41149.1455702343@critter.freebsd.dk> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 09:45:53 -0000 -------- In message <20160217094241.GF91220@kib.kiev.ua>, Konstantin Belousov writes: >E.g. there might be no interrupts set up yet, and idle thread on UP >gets on CPU, then the whole boot process could deadlock. idle_thread: while (!interrupts_setup_done()) yield(); -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@freebsd.org Wed Feb 17 09:46:53 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68BA3AAB67E for ; Wed, 17 Feb 2016 09:46:53 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5795017C5 for ; Wed, 17 Feb 2016 09:46:53 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: by mailman.ysv.freebsd.org (Postfix) id 56CFCAAB67D; Wed, 17 Feb 2016 09:46:53 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 567FCAAB67C for ; Wed, 17 Feb 2016 09:46:53 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 22F4817C1; Wed, 17 Feb 2016 09:46:53 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id 5B5F84F865; Wed, 17 Feb 2016 09:46:52 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTP id u1H9kpcH041170; Wed, 17 Feb 2016 09:46:52 GMT (envelope-from phk@phk.freebsd.dk) To: Warner Losh cc: Julian Elischer , "freebsd-arch@freebsd.org" Subject: Re: Starting APs earlier during boot In-reply-to: From: "Poul-Henning Kamp" References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <56C4061B.6010601@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <41168.1455702411.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Wed, 17 Feb 2016 09:46:51 +0000 Message-ID: <41169.1455702411@critter.freebsd.dk> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 09:46:53 -0000 -------- In message , Warner Losh writes: >> what is the goal? cleaner code? faster boot? > >Two goals were in his original email. And I hope that in the longer term we also aim to configure I/O in parallel ? -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-arch@freebsd.org Wed Feb 17 16:14:36 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 524DFAAB2D9 for ; Wed, 17 Feb 2016 16:14:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 31DFF1A93 for ; Wed, 17 Feb 2016 16:14:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 2EBB2AAB2D5; Wed, 17 Feb 2016 16:14:36 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 148EAAAB2D4 for ; Wed, 17 Feb 2016 16:14:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD18B1A91 for ; Wed, 17 Feb 2016 16:14:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qg0-x233.google.com with SMTP id b35so15250253qge.0 for ; Wed, 17 Feb 2016 08:14:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=m67pINLJh1ccDUkvPpk91/htD9PORq8X3jApx2eOuLE=; b=qkVkhwZNMOsNJ1TvlhWzU/wQstUg4iLCgQidLmP81e7ex6rE25T699HqXN75uSY8jI vyvswSdlbJKI0Twt1pEnypacPBjvdSvL1wmxpKetmhfwvsjlp8Bp5X1R4QwefPa3xgUV g9f+jNmOtWSK7R+ds5Se/wiUvxU6ocdwKGLfRVypiTNqkRWuU8lCMM+AsNQAXtwM1Nwn BRtSVt9qTJ70f0kCb1Uez673KPrem8pMW2xF3rc2CSyc3A0McmCeeK9BKSxl01RjxqSt tTXMun6Zo+ioa9ipcCRMDsGvFjG6e23jy0tL0i1JFv3+C+BEx4lUEQV/hZuSS3QwzEg6 sMsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=m67pINLJh1ccDUkvPpk91/htD9PORq8X3jApx2eOuLE=; b=L/UQK3g+mAD6ZdsVrEgGldKaH0hU5bQ8tieO+70BEAgPdc8YKUrC8XfGIpB8vpu1w2 H0rbMP6r2Dh4RbDg3fEl/U5Y2qWlTo/TD6MQg8aUDOwcMqWO/BTqogX+tAgS8RpsOJ6S jRoRoZGHYSkwu+Z0A2JuFoYCsxkmxTGnejVR8IH0HQn9BD4vToujv9hUm5VixhGOkPa2 AllPYYFTsLPUsccpvuA/pf3kVR7zkveSg+nMeS9bJMyHKi6lgMSB9B7IKXktcVnOLl56 oLbS35uBJO/+Ui4FGYqPflUsAP4GP59w87fW8I5LS+J3IaXjVZmoYcmtOeK1pRxnPJBs ZhwQ== X-Gm-Message-State: AG10YOSIfNILPgZz16xkNHceTmXRvbyiAMTZx5Jja5fnzb79cODYWgjwkgLe6jRXUR7oyJF4IpEgewpam1q6YQ== MIME-Version: 1.0 X-Received: by 10.140.135.147 with SMTP id 141mr3084943qhh.74.1455725674757; Wed, 17 Feb 2016 08:14:34 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.30.166 with HTTP; Wed, 17 Feb 2016 08:14:34 -0800 (PST) X-Originating-IP: [69.53.245.31] In-Reply-To: <41169.1455702411@critter.freebsd.dk> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <56C4061B.6010601@freebsd.org> <41169.1455702411@critter.freebsd.dk> Date: Wed, 17 Feb 2016 09:14:34 -0700 X-Google-Sender-Auth: 3rFLD129xyZOCl91SrlBbqUcyqg Message-ID: Subject: Re: Starting APs earlier during boot From: Warner Losh To: Poul-Henning Kamp Cc: Julian Elischer , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 16:14:36 -0000 On Wed, Feb 17, 2016 at 2:46 AM, Poul-Henning Kamp wrote: > -------- > In message KYPxDcs5jS2iDW-OowwgoFL3Q@mail.gmail.com>, Warner Losh writes: > > >> what is the goal? cleaner code? faster boot? > > > >Two goals were in his original email. > > And I hope that in the longer term we also aim to configure I/O > in parallel ? What do you mean by 'configure I/O in parallel?' Warner > From owner-freebsd-arch@freebsd.org Wed Feb 17 16:16:24 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F4C1AAB430 for ; Wed, 17 Feb 2016 16:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 561841CBB for ; Wed, 17 Feb 2016 16:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 51684AAB42E; Wed, 17 Feb 2016 16:16:24 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50F46AAB42D; Wed, 17 Feb 2016 16:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C44B11CB7; Wed, 17 Feb 2016 16:16:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1HGGC04088915 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 17 Feb 2016 18:16:13 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1HGGC04088915 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1HGGCOq088914; Wed, 17 Feb 2016 18:16:12 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 17 Feb 2016 18:16:12 +0200 From: Konstantin Belousov To: Martin Simmons Cc: vangyzen@FreeBSD.org, threads@FreeBSD.org, arch@FreeBSD.org Subject: Re: libthr shared locks Message-ID: <20160217161612.GL91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <20160213143815.GB91220@kib.kiev.ua> <201602151417.u1FEHKwL003392@higson.cam.lispworks.com> <20160215144410.GT91220@kib.kiev.ua> <201602151735.u1FHZXKV006190@higson.cam.lispworks.com> <20160215175621.GU91220@kib.kiev.ua> <201602161617.u1GGHkil023634@higson.cam.lispworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201602161617.u1GGHkil023634@higson.cam.lispworks.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 16:16:24 -0000 On Tue, Feb 16, 2016 at 04:17:46PM +0000, Martin Simmons wrote: > >>>>> On Mon, 15 Feb 2016 19:56:21 +0200, Konstantin Belousov said: > > > > One process which executed pthread_barrier_init(), performed what you > > proposed. What should do the pthread_barrier_wait() call in another > > process, which shares the 'barrier' with the first process, but does > > not share the whole address space ? After your pthread_barrier_init() > > executed, barrier contains the address of the object (off-page) in the > > other address space, for that process. > > Ah, sorry, I understand now (the init functions are called before any > sharing). Well, the memory is either already shared between processes, or it should become shared before other process may operate on the object carried by the memory. It is that pthread_mutex_init() must be called before any other pthread_mutex_*() functions, but only once globally. > > How should the destroy functions be used by the processes? I.e. should only > the "last" process call destroy or can every process call it? Hm, this is a good observation. pthread_mutex_destroy() must be called only by last process, i.e. no other pthread_mutex_* calls are allowed for the object after the _destroy() was called. What this means for my implementation, is that processes other than the _destroy() caller keep the record in the pshared cache, and this needs fixing. For now, I added a mechanism which scans the whole hash and re-checks the validity on any object destroy. This can be optimized, e.g. by doing the scan only each N calls to _destroy(), or by scanning only the same hash chain, or by not doing this at all. I think it is an acceptable compromise for now. A specific test for the case is at https://www.kib.kiev.ua/kib/pshared/pthread_shared_destroy.c Updated patch https://www.kib.kiev.ua/kib/pshared/pshared.4.patch From owner-freebsd-arch@freebsd.org Wed Feb 17 16:45:48 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A010EAAA2EC for ; Wed, 17 Feb 2016 16:45:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8A0AB1D5B for ; Wed, 17 Feb 2016 16:45:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 85326AAA2E8; Wed, 17 Feb 2016 16:45:48 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84AABAAA2E7; Wed, 17 Feb 2016 16:45:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 035231D59; Wed, 17 Feb 2016 16:45:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1HGjfC6095470 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 17 Feb 2016 18:45:41 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1HGjfC6095470 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1HGjfUg095469; Wed, 17 Feb 2016 18:45:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 17 Feb 2016 18:45:41 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20160217164541.GM91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 16:45:48 -0000 On Tue, Feb 16, 2016 at 12:27:12PM -0500, Daniel Eischen wrote: > On Tue, 16 Feb 2016, Konstantin Belousov wrote: > > > On Mon, Feb 15, 2016 at 03:39:18PM -0600, Eric van Gyzen wrote: > >> My only comment on kern_umtx.c is, why are the permission checks compiled out? > > > > Assume that we changed the ABI of libthr and shared locks do not require > > an offpage. > > How are you coming with that? I thought maybe you were going to think > about making the synch types structs, but not actually changing the > implementation (yet). Are you asking what is the state of the work for changing the libthr ABI by replacing object pointers with inline structures, am I reading the question right ? (Sorry, english is not my native language) I do plan to introduce inlined objects (most likely in the form of libthr2 initially, i.e. cc -D_LIBTHR2 -o file file.c -lthr2). But my plans are to get the existing patch for pshared into the tree for 11.0. After that I wanted to implement robust mutexes, still in the context of the libthr. Then libthr2. Finishing the current patch for 11.0 is the immediate goal, while I did not forget neither abandoned the inline alternative and do plan to work on it. From owner-freebsd-arch@freebsd.org Wed Feb 17 17:38:00 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7909FAAB98A for ; Wed, 17 Feb 2016 17:38:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 65B011A90 for ; Wed, 17 Feb 2016 17:38:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 60E9EAAB986; Wed, 17 Feb 2016 17:38:00 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60709AAB985; Wed, 17 Feb 2016 17:38:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0FC891A8D; Wed, 17 Feb 2016 17:37:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id u1HHbqJV029498; Wed, 17 Feb 2016 12:37:52 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Wed, 17 Feb 2016 12:37:52 -0500 (EST) Date: Wed, 17 Feb 2016 12:37:52 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20160217164541.GM91220@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 17:38:00 -0000 On Wed, 17 Feb 2016, Konstantin Belousov wrote: > On Tue, Feb 16, 2016 at 12:27:12PM -0500, Daniel Eischen wrote: >> On Tue, 16 Feb 2016, Konstantin Belousov wrote: >> >>> On Mon, Feb 15, 2016 at 03:39:18PM -0600, Eric van Gyzen wrote: >>>> My only comment on kern_umtx.c is, why are the permission checks compiled out? >>> >>> Assume that we changed the ABI of libthr and shared locks do not require >>> an offpage. >> >> How are you coming with that? I thought maybe you were going to think >> about making the synch types structs, but not actually changing the >> implementation (yet). > > Are you asking what is the state of the work for changing the libthr > ABI by replacing object pointers with inline structures, am I reading > the question right ? (Sorry, english is not my native language) I am sorry, my question was not phrased very well. Sometimes I am amazed that non-native English speakers can write better English than native English speakers :-) But, yes, that was my question. > I do plan to introduce inlined objects (most likely in the form of > libthr2 initially, i.e. cc -D_LIBTHR2 -o file file.c -lthr2). But my > plans are to get the existing patch for pshared into the tree for 11.0. > After that I wanted to implement robust mutexes, still in the context of > the libthr. Then libthr2. As soon as this is done, will we build FreeBSD base OS against the new API? So only ports would be affected. I was hoping to see partially inlined objects (with the first word being a pointer to the allocated object, just like now) in for 11.0. So by 12.0 we could make the switch over to fully inlined. If we introduce either partially or fully inlined objects for 11.1 (with your libthr2), how does that affect our ability to move to fully inlined by default (base & ports) for 12.0? We would not have had a full release cycle for the ABI change. > Finishing the current patch for 11.0 is the immediate goal, while I did > not forget neither abandoned the inline alternative and do plan to work > on it. Thanks! -- DE From owner-freebsd-arch@freebsd.org Wed Feb 17 17:39:27 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8AD3AABAD8 for ; Wed, 17 Feb 2016 17:39:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A781F1BF7 for ; Wed, 17 Feb 2016 17:39:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A67F4AABAD5; Wed, 17 Feb 2016 17:39:27 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6158AABAD0 for ; Wed, 17 Feb 2016 17:39:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 88B6C1BF6 for ; Wed, 17 Feb 2016 17:39:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 25636B918; Wed, 17 Feb 2016 12:39:26 -0500 (EST) From: John Baldwin To: Konstantin Belousov Cc: arch@freebsd.org Subject: Re: Starting APs earlier during boot Date: Wed, 17 Feb 2016 09:00:26 -0800 Message-ID: <38395751.7hkq6XTlKk@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160217094241.GF91220@kib.kiev.ua> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <20160217094241.GF91220@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 17 Feb 2016 12:39:26 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 17:39:27 -0000 On Wednesday, February 17, 2016 11:42:41 AM Konstantin Belousov wrote: > On Tue, Feb 16, 2016 at 12:50:22PM -0800, John Baldwin wrote: > > However, we can run a slightly simpler version of our scheduler before > > timers are working. In fact, sleep/wakeup work just fine fairly early (we > > allocate the necessary structures at SI_SUB_KMEM which is before the APs > > are even started). Once idle threads are created and ready we could in > > theory let the APs startup and run other threads. You just don't have working > > timeouts. OTOH, you can sort of simulate timeouts if you modify the scheduler > > to yield the CPU instead of blocking the thread for a sleep with a timeout. > > The effect would be for threads that do sleeps with a timeout to fall back to > > polling before timers are working. In practice, all of the early kernel > > threads use sleeps without timeouts when idle so this doesn't really matter. > I understand that timeouts can be somewhat simulated this way. > > But I do not quite understand how generic scheduling can work without > (timer) interrupts. Suppose that we have two threads 1 and 2 of the same > priority, both runnable, and due to some event thread 2 preempted thread > 1. If thread 2 just runs without calling the preempt functions like > msleep, what would guarentee that thread 1 eventually gets it CPU slice ? Nothing, but the only thread we have that does that during early startup is thread0 (which is what should be running as it is the one that makes progress to getting timers setup). Currently the sleep calls just always return which means if any thread gets on the CPU it never yields and is stuck forever. My changes make it so only a CPU bound thread is stuck forever, and we only have one such thread before timers are running: thread0. > E.g. there might be no interrupts set up yet, and idle thread on UP > gets on CPU, then the whole boot process could deadlock. The idle threads are special as they yield explicitly if there are any runnable threads on the run queues. -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Feb 17 19:15:59 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9AD62AABCAD for ; Wed, 17 Feb 2016 19:15:59 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 890CB1CA7 for ; Wed, 17 Feb 2016 19:15:59 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: by mailman.ysv.freebsd.org (Postfix) id 881C7AABCAC; Wed, 17 Feb 2016 19:15:59 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87BB2AABCAB for ; Wed, 17 Feb 2016 19:15:59 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 53C0E1CA6; Wed, 17 Feb 2016 19:15:59 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id CB60A4F865; Wed, 17 Feb 2016 19:15:57 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTP id u1HJFunV042922; Wed, 17 Feb 2016 19:15:57 GMT (envelope-from phk@phk.freebsd.dk) To: Warner Losh cc: Julian Elischer , "freebsd-arch@freebsd.org" Subject: Re: Starting APs earlier during boot In-reply-to: From: "Poul-Henning Kamp" References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <56C4061B.6010601@freebsd.org> <41169.1455702411@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <42920.1455736556.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Wed, 17 Feb 2016 19:15:56 +0000 Message-ID: <42921.1455736556@critter.freebsd.dk> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 19:15:59 -0000 -------- In message , Warner Losh writes: >> And I hope that in the longer term we also aim to configure I/O >> in parallel ? > >What do you mean by 'configure I/O in parallel?' probe/attach device drivers in parallel to speed up boot. -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-arch@freebsd.org Wed Feb 17 19:20:57 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8129AAC01E for ; Wed, 17 Feb 2016 19:20:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 896B01268 for ; Wed, 17 Feb 2016 19:20:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 2933AB918; Wed, 17 Feb 2016 14:20:56 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Cc: Poul-Henning Kamp , Warner Losh Subject: Re: Starting APs earlier during boot Date: Wed, 17 Feb 2016 10:19:43 -0800 Message-ID: <7708748.E0vxJ7C8cf@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <41169.1455702411@critter.freebsd.dk> References: <1730061.8Ii36ORVKt@ralph.baldwin.cx> <41169.1455702411@critter.freebsd.dk> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 17 Feb 2016 14:20:56 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Feb 2016 19:20:57 -0000 On Wednesday, February 17, 2016 09:46:51 AM Poul-Henning Kamp wrote: > -------- > In message , Warner Losh writes: > > >> what is the goal? cleaner code? faster boot? > > > >Two goals were in his original email. > > And I hope that in the longer term we also aim to configure I/O > in parallel ? I'm a bit leery of doing this fully parallel. In particular, users currently depend on the behavior of deterministic names in new-bus (so em0 is always em0 and not sometimes em1). OTOH, I think that we could eventually allow drivers to start doing some of the background scans sooner and only harvest the results at the interrupt config hooks instead of starting the scans and timers at the interrupt config hook (and this is a step towards that). From what I understand, most of our boot time start up delay isn't the new-bus device probe but userland startup. Nevertheless, I think the changes I've proposed here are a prerequisite for even thinking about possibly making device probe more parallel. -- John Baldwin From owner-freebsd-arch@freebsd.org Thu Feb 18 15:33:01 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF5F5AAD0EA for ; Thu, 18 Feb 2016 15:33:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B3C83E74 for ; Thu, 18 Feb 2016 15:33:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B0E9CAAD0E8; Thu, 18 Feb 2016 15:33:01 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96892AAD0E6; Thu, 18 Feb 2016 15:33:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3EEBEE6E; Thu, 18 Feb 2016 15:33:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1IFWuT4025136 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 18 Feb 2016 17:32:56 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1IFWuT4025136 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1IFWu76025135; Thu, 18 Feb 2016 17:32:56 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 18 Feb 2016 17:32:56 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20160218153256.GS91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Feb 2016 15:33:02 -0000 On Wed, Feb 17, 2016 at 12:37:52PM -0500, Daniel Eischen wrote: > On Wed, 17 Feb 2016, Konstantin Belousov wrote: > > I do plan to introduce inlined objects (most likely in the form of > > libthr2 initially, i.e. cc -D_LIBTHR2 -o file file.c -lthr2). But my > > plans are to get the existing patch for pshared into the tree for 11.0. > > After that I wanted to implement robust mutexes, still in the context of > > the libthr. Then libthr2. > > As soon as this is done, will we build FreeBSD base OS against > the new API? So only ports would be affected. I do not think this is feasible. Base system does not have any significant thread consumers, might be only ntpd qualifies. Small things like ngctl are not that important. But base system provides C++ runtime for ports and I suspect that libc++ depends on the libthr ABI. Even jemalloc depends on libthr ABI. So changing only the base ABI is probably impossible, from the first look the switch like WITH_LIBTHR2_DEFAULT would be a flag day. Anyway, this must be considered carefully during the later stage of the libthr2 development, right now it is rather empty speculation on my side. > > I was hoping to see partially inlined objects (with the first word > being a pointer to the allocated object, just like now) in for 11.0. > So by 12.0 we could make the switch over to fully inlined. I do not see how this is realistic. The issue of introducing the inlined objects is the stumbling block, regardless of their use in full or only as a placeholder for the pointer. Not even mentioning that we do not want to radically break the userspace ABI for 11.0 when the freeze is so close, we would need much more time to develop the WITH_LIBTHR2 alone. This is my opinion, but it is supported by my experience. > > If we introduce either partially or fully inlined objects for 11.1 > (with your libthr2), how does that affect our ability to move to > fully inlined by default (base & ports) for 12.0? We would not > have had a full release cycle for the ABI change. I cannot answer this question, the issue requires careful investigation, as I noted earlier in the message. Might be I or somebody else would be able to produce some trick that would ease the pain. If not, the bump of dso version numbers and dynamic linker awareness of the incompatible libraries would be to coded. > > > Finishing the current patch for 11.0 is the immediate goal, while I did > > not forget neither abandoned the inline alternative and do plan to work > > on it. That is. From owner-freebsd-arch@freebsd.org Thu Feb 18 17:10:01 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD176AAC72B for ; Thu, 18 Feb 2016 17:10:01 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B093DD44 for ; Thu, 18 Feb 2016 17:10:01 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id ACDC1AAC728; Thu, 18 Feb 2016 17:10:01 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC613AAC727; Thu, 18 Feb 2016 17:10:01 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 725C9D34; Thu, 18 Feb 2016 17:10:01 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id u1IH9xD2034133; Thu, 18 Feb 2016 12:09:59 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Thu, 18 Feb 2016 12:09:59 -0500 (EST) Date: Thu, 18 Feb 2016 12:09:59 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20160218153256.GS91220@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> <20160218153256.GS91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Feb 2016 17:10:01 -0000 On Thu, 18 Feb 2016, Konstantin Belousov wrote: > On Wed, Feb 17, 2016 at 12:37:52PM -0500, Daniel Eischen wrote: >> On Wed, 17 Feb 2016, Konstantin Belousov wrote: >>> I do plan to introduce inlined objects (most likely in the form of >>> libthr2 initially, i.e. cc -D_LIBTHR2 -o file file.c -lthr2). But my >>> plans are to get the existing patch for pshared into the tree for 11.0. >>> After that I wanted to implement robust mutexes, still in the context of >>> the libthr. Then libthr2. >> >> As soon as this is done, will we build FreeBSD base OS against >> the new API? So only ports would be affected. > I do not think this is feasible. Base system does not have any significant > thread consumers, might be only ntpd qualifies. Small things like ngctl > are not that important. > > But base system provides C++ runtime for ports and I suspect that libc++ > depends on the libthr ABI. Even jemalloc depends on libthr ABI. So > changing only the base ABI is probably impossible, from the first look > the switch like WITH_LIBTHR2_DEFAULT would be a flag day. Anyway, this > must be considered carefully during the later stage of the libthr2 > development, right now it is rather empty speculation on my side. I would think partially inlined objects (still a pointer inside) could be made in libthr (not libthr2) by default for 11.0 (or 11.x). So that the only change is the size of the objects, not anything else. The only breakage would be layout related. -- DE From owner-freebsd-arch@freebsd.org Fri Feb 19 00:33:02 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6C08AABBE7 for ; Fri, 19 Feb 2016 00:33:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BFD7B145D for ; Fri, 19 Feb 2016 00:33:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id BC98FAABBE4; Fri, 19 Feb 2016 00:33:02 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC172AABBE3; Fri, 19 Feb 2016 00:33:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DA7B145B; Fri, 19 Feb 2016 00:33:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1J0Wted021594 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 19 Feb 2016 02:32:55 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1J0Wted021594 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1J0Wtdm021593; Fri, 19 Feb 2016 02:32:55 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 19 Feb 2016 02:32:55 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20160219003255.GU91220@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> <20160218153256.GS91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2016 00:33:03 -0000 On Thu, Feb 18, 2016 at 12:09:59PM -0500, Daniel Eischen wrote: > On Thu, 18 Feb 2016, Konstantin Belousov wrote: > > But base system provides C++ runtime for ports and I suspect that libc++ > > depends on the libthr ABI. Even jemalloc depends on libthr ABI. So > > changing only the base ABI is probably impossible, from the first look > > the switch like WITH_LIBTHR2_DEFAULT would be a flag day. Anyway, this > > must be considered carefully during the later stage of the libthr2 > > development, right now it is rather empty speculation on my side. This paragraph is relevant for my answer below. > > I would think partially inlined objects (still a pointer inside) > could be made in libthr (not libthr2) by default for 11.0 (or 11.x). > So that the only change is the size of the objects, not anything > else. The only breakage would be layout related. Structure size is part of the library ABI, when the structure is exposed to user, it cannot be increased without consequences. Changing the size of the libthr objects changes layout of the objects embedding the locks. Doing class MyLock { private: pthread_mutex_t m_lock; const char *name; }; is the common and very popular practice. With the change proposed to libthr.so.3, you get subtle and sudden ABI breakage for all libthr consumers as well. In particular, it would affect libc (jemalloc) and libc++. Depending on the variant of headers used for the compilation, you get different layout for user structures. From owner-freebsd-arch@freebsd.org Fri Feb 19 02:31:52 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 170DCAAD5F8 for ; Fri, 19 Feb 2016 02:31:52 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 026BF1FB5 for ; Fri, 19 Feb 2016 02:31:52 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 017AFAAD5F6; Fri, 19 Feb 2016 02:31:52 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F4186AAD5F5; Fri, 19 Feb 2016 02:31:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E9D21FB2; Fri, 19 Feb 2016 02:31:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id u1J2VibS010379; Thu, 18 Feb 2016 21:31:44 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Thu, 18 Feb 2016 21:31:44 -0500 (EST) Date: Thu, 18 Feb 2016 21:31:44 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20160219003255.GU91220@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> <20160218153256.GS91220@kib.kiev.ua> <20160219003255.GU91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2016 02:31:52 -0000 On Fri, 19 Feb 2016, Konstantin Belousov wrote: > On Thu, Feb 18, 2016 at 12:09:59PM -0500, Daniel Eischen wrote: >> On Thu, 18 Feb 2016, Konstantin Belousov wrote: >>> But base system provides C++ runtime for ports and I suspect that libc++ >>> depends on the libthr ABI. Even jemalloc depends on libthr ABI. So >>> changing only the base ABI is probably impossible, from the first look >>> the switch like WITH_LIBTHR2_DEFAULT would be a flag day. Anyway, this >>> must be considered carefully during the later stage of the libthr2 >>> development, right now it is rather empty speculation on my side. > This paragraph is relevant for my answer below. > >> >> I would think partially inlined objects (still a pointer inside) >> could be made in libthr (not libthr2) by default for 11.0 (or 11.x). >> So that the only change is the size of the objects, not anything >> else. The only breakage would be layout related. > > Structure size is part of the library ABI, when the structure is exposed > to user, it cannot be increased without consequences. > > Changing the size of the libthr objects changes layout of the objects > embedding the locks. Doing > class MyLock { > private: > pthread_mutex_t m_lock; > const char *name; > }; > is the common and very popular practice. With the change proposed to > libthr.so.3, you get subtle and sudden ABI breakage for all libthr > consumers as well. In particular, it would affect libc (jemalloc) and > libc++. Depending on the variant of headers used for the compilation, > you get different layout for user structures. libc is part of FreeBSD, so it would be recompiled for the new size. I was also assuming library version bumps. -- DE From owner-freebsd-arch@freebsd.org Fri Feb 19 15:08:37 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D730AAADA40 for ; Fri, 19 Feb 2016 15:08:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BD1BF139B for ; Fri, 19 Feb 2016 15:08:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B9015AADA3C; Fri, 19 Feb 2016 15:08:37 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EB5FAADA3B; Fri, 19 Feb 2016 15:08:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4702C1399; Fri, 19 Feb 2016 15:08:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u1JF8Sei034403 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 19 Feb 2016 17:08:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u1JF8Sei034403 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u1JF8RRu034402; Fri, 19 Feb 2016 17:08:27 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 19 Feb 2016 17:08:27 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: Eric van Gyzen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20160219150827.GW91220@kib.kiev.ua> References: <56BE69B8.9020808@FreeBSD.org> <56C24586.9050906@FreeBSD.org> <20160216113222.GY91220@kib.kiev.ua> <20160217164541.GM91220@kib.kiev.ua> <20160218153256.GS91220@kib.kiev.ua> <20160219003255.GU91220@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2016 15:08:38 -0000 On Thu, Feb 18, 2016 at 09:31:44PM -0500, Daniel Eischen wrote: > On Fri, 19 Feb 2016, Konstantin Belousov wrote: > > > On Thu, Feb 18, 2016 at 12:09:59PM -0500, Daniel Eischen wrote: > >> On Thu, 18 Feb 2016, Konstantin Belousov wrote: > >>> But base system provides C++ runtime for ports and I suspect that libc++ > >>> depends on the libthr ABI. Even jemalloc depends on libthr ABI. So > >>> changing only the base ABI is probably impossible, from the first look > >>> the switch like WITH_LIBTHR2_DEFAULT would be a flag day. Anyway, this > >>> must be considered carefully during the later stage of the libthr2 > >>> development, right now it is rather empty speculation on my side. > > This paragraph is relevant for my answer below. > > > >> > >> I would think partially inlined objects (still a pointer inside) > >> could be made in libthr (not libthr2) by default for 11.0 (or 11.x). > >> So that the only change is the size of the objects, not anything > >> else. The only breakage would be layout related. > > > > Structure size is part of the library ABI, when the structure is exposed > > to user, it cannot be increased without consequences. > > > > Changing the size of the libthr objects changes layout of the objects > > embedding the locks. Doing > > class MyLock { > > private: > > pthread_mutex_t m_lock; > > const char *name; > > }; > > is the common and very popular practice. With the change proposed to > > libthr.so.3, you get subtle and sudden ABI breakage for all libthr > > consumers as well. In particular, it would affect libc (jemalloc) and > > libc++. Depending on the variant of headers used for the compilation, > > you get different layout for user structures. > > libc is part of FreeBSD, so it would be recompiled for the new > size. I was also assuming library version bumps. Increasing the structures sizes (and bumping versions) would - break ABI - invalidate all the work which was done by people from the FreeBSD 7.x time to keep the ABI stable, by writing shims, by adopting changes to provide compatibility, and by testing the compatibility without any benefit of a new feature. It would be only promised to provide a new feature sometime in the 11.x scope. My immediate goal is to get the published patch committed. After that, I want to look at the implementation of the robust mutexes. I hope that this would be done for 11.0, but if not, it should be mergeable. The libthr2 stuff, by which I call everything related to inlining, should be started after that. From owner-freebsd-arch@freebsd.org Fri Feb 19 23:06:58 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D2A6AAE697 for ; Fri, 19 Feb 2016 23:06:58 +0000 (UTC) (envelope-from dumbbell@FreeBSD.org) Received: from mail.made4.biz (mail.made4.biz [IPv6:2001:41d0:2:c018::1:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C67D1F45 for ; Fri, 19 Feb 2016 23:06:58 +0000 (UTC) (envelope-from dumbbell@FreeBSD.org) Received: from [176.158.145.63] (helo=magellan.dumbbell.fr) by mail.made4.biz with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aWu8V-0006Qu-Ve for freebsd-arch@FreeBSD.org; Sat, 20 Feb 2016 00:06:56 +0100 To: freebsd-arch@FreeBSD.org From: =?UTF-8?Q?Jean-S=c3=a9bastien_P=c3=a9dron?= Subject: "initial-exec" TLS model and dlopen(3) X-Enigmail-Draft-Status: N1110 Message-ID: <56C7A00A.3030802@FreeBSD.org> Date: Sat, 20 Feb 2016 00:06:50 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="GXRNBuFSWdni5fbROsATE9SqfAa5CesF8" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2016 23:06:58 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --GXRNBuFSWdni5fbROsATE9SqfAa5CesF8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello! =3D=3D Context =3D=3D After Mesa 11.2.0 branch point (RC1 is scheduled today), Emil Velikov, Mesa release engineer, told us he plans to only keep the TLS-based GL dispatcher and remove the other code path. However, even if all Linux distributions use the TLS-based one for a long time, it was still turned off by default in the configure, and we never field-tested it on FreeBSD. We enabled the --enable-glx-tls flag in Mesa in our development Ports tree. Unfortunately, some applications segfault after that. Firefox is one of them. Below is what I understand about this issue so far. =3D=3D The issue =3D=3D Most applications are linked directly to libGL.so. In this case, there is no problem. For example, glxgears is linked directly to libGL.so. Some of them use dlopen(3) to directly or indirectly load libGL.so. For example, Firefox dlopens libxul.so which is linked to libGL.so. Mesa uses the "initial-tls" model for at least one TLS variable: https://cgit.freedesktop.org/mesa/mesa/tree/src/glx/glxcurrent.c#n79 I'm new to TLS implementation details, but if I understand correctly, this model is a static one, meaning that a variable address is known and it's accessed directly, like a normal variable, as opposed to dynamic TLS models where a variable address is first queried with __tls_get_addr(). This is all transparent to the program because the compiler is responsible for generating the appropriate code, depending on the model. In the case of a direct link like glxgears, our rtld (ld-elf.so) allocates space during startup to copy static TLS variables from the program and all linked libraries. libGL.so finds its variables where it expects them to be, glxgears is happy. In the case of a dlopen(3) like Firefox, our rtld maps the dlopen'd object and all its linked libraries but it doesn't look for any static TLS variables. libGL.so accesses the allocated TLS storage (there is a small extra chunk of zero'd memory allocated) but its variables were not copied. So it gets a NULL pointer, dereferences it, End of the World. Here is a small test program to demonstrate the crash: https://github.com/dumbbell/test-tls-initial-exec =3D=3D Solutions =3D=3D A first workaround is to LD_PRELOAD libGL.so or link the program directly to libGL.so. Another solution is the following: in the Glibc (quite popular these days), they allocate extra static TLS space beside the size of the TLS variables available at startup (ie. TLS variables from the program and linked libraries). Then, when a library is dlopen'd with static TLS variables, they are copied to this extra space. This space is not dynamically extended, so first loaded, first served. If there is no space left, I think dlopen(3) fails. In FreeBSD's rtld, we already allocate extra space. See for instance the use of the RTLD_STATIC_TLS_EXTRA constant here: https://github.com/freebsd/freebsd/blob/master/libexec/rtld-elf/amd64/rel= oc.c#L453-L464 The command even says that this extra space is allocated specifically for dynamic modules. However, I don't see where we use this space. dlopen_object() doesn't mess with TLS at all (or I'm missing something). FWIW, Mesa's libGL.so is not the only one to do this. NVIDIA's libGL.so uses the same technic and apparently, AMD Catalyst's one too on Linux. I wonder if the following bug is caused by this exact issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205149 I would like to modify dlopen_object() to install static TLS variables in this extra space. Or do you suggest a better alternative? If possible, I would like to have this into FreeBSD 10.3-RELEASE to avoid future maintenance headaches of Mesa. Some references about this issue: https://bugs.freedesktop.org/show_bug.cgi?id=3D35268 http://www.redhat.com/archives/phil-list/2003-February/msg00077.html https://gcc.gnu.org/ml/gcc/2015-02/msg00095.html --=20 Jean-S=C3=A9bastien P=C3=A9dron --GXRNBuFSWdni5fbROsATE9SqfAa5CesF8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWx6APXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2NzA4N0ZEMUFFQUUwRTEyREJDNkE2RjAz OUU5OTc2MUE1RkQ5NENDAAoJEDnpl2Gl/ZTM9fkP/2NdJUX7n6x9bD67dJ/tGQRO 9PLAoChaK0nnwwPCHa9UsRvHNSDLHYqwmdyWugzsfNn52gNu3v1tDFFFcqEBEi0j bKihBwFq+sCbwbQ06L/aKhXFCv4RMHN9GerOSD6KRn1e90enn0r5Oy3Wmk2RoOaW 653abIZPSr7xkh0KiiZM4UL8VVgsyzKwSCHOoBbwqyMhiDFOD7a6M0ix0xgVoyKT PUcG69tzks9zzqrLjTWeG8B6Asc+mtJoqYGk5fSywI5ni6UbNJuFBuK7gORymoPw kaLJMpWk7IcmFmxJ3U9NaKZy5qn9+R2MLiULk98X/Dnz+lxs/uYpIj9iKsYou7PJ XoCMvGVBLxm9Y+OrvY3rx8PQ2zfHbGDza07qHzOtzfXIdi/Zb+QuOfA+ZcACyngf MFiLVkSIoA/KpaMnmMgPNWY+IbTY827O5WoGtds2BH6kRyx4r/PdTfC84muoIABC MG/Wiv26jILcWbyvAm7AVr5Nzl/aLSIBfYix2Jiy2JKAXRvN1uNrr0To6PYt4RmL jHB8AlGrw3ZU6tJPwGznKfHSagG+EnoOHdxAYSwUkLqzmP35MIEMJJWv6xzONQnh 6zml58gRBECU6YcQGCdeQ680Rjqx0rcZrNF8setjB111gHSNYPI0v6LYXL4kH/a5 lYH9D5e/LCdtu5sW7z6o =Xs3Z -----END PGP SIGNATURE----- --GXRNBuFSWdni5fbROsATE9SqfAa5CesF8-- From owner-freebsd-arch@freebsd.org Sat Feb 20 04:01:08 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17E56AAF3F8 for ; Sat, 20 Feb 2016 04:01:08 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C54711C7E; Sat, 20 Feb 2016 04:01:07 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: by mail-qg0-x235.google.com with SMTP id y89so76949611qge.2; Fri, 19 Feb 2016 20:01:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type; bh=h/Nby7Nc57jtVPurnnmy0fpfbQD+PoXx4lvyi9hV4SA=; b=gZEdByEmIHxCR2vOs8/8dztY9irEXkhltaSGGpBTGiPS7ifLbPgvtDCk1EomapfSE0 cFV9XVvPBwtircy3SuaLL/pvK8RW3qD0O1f3wSK/HkdbBmsB3K4gtegeeDHosbjotKTv 7JgXVx09zDJTqJLozExXIFmQTQxHG0FGGB+luxJPfw7hiXNDNrm+VAo7bd9Uw6ANU/AH 7lBhAQxh5Z+80rUbHKwtKiWbv8s42RnZ0bRWFn22YddEbe3UvgpvDs/7XIjp4cwCi3Sm lQkvwF0zUZZFoxxcMn8K+8senEy86lR+U9UvcyMmn0FKaRMT7x+mKnNl9M8ttOBdBs8O a+GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type; bh=h/Nby7Nc57jtVPurnnmy0fpfbQD+PoXx4lvyi9hV4SA=; b=gdwzlenkzeR8CVBKiOHK8fCGtiOHSULnWqAnLQf23BJDRgeiQrQJOLqmxs8uA3mO9m bfW+Eu0Uh6DZkeCJbMMZsUzD35Nsw4qaJEo9+mjzHxTZFa9HKUaso6Il8eMMcNb0sfRw wHxouXb1oEB4xT5ZCO/AvYj13QkHEv/uyysi+z5w/Ul/ICtXue19Crbbtk6iR19ypXzY 59clT1Tm93VFq0m8rDanjXUz+GCNmkDD1569/zWOJZKcqfHqzRj97+lF0BpTVF/Pu61f SaWdW7J3zThtWJoPhrKWZEurTQO7Z3JwATnGnySp0M739gV2YKWwM1V9qpJSef3zXTdF XHmA== X-Gm-Message-State: AG10YOTp9eKkSROOpK6g+74yUUd+wEqrrLOFBWp71mcdbyQOqhEMdttoOZNwjCt6qpTsWQ== X-Received: by 10.140.102.14 with SMTP id v14mr20468674qge.48.1455940866873; Fri, 19 Feb 2016 20:01:06 -0800 (PST) Received: from kan ([2601:18f:802:4680:226:18ff:fe00:232e]) by smtp.gmail.com with ESMTPSA id c191sm5952541qhc.12.2016.02.19.20.01.05 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 19 Feb 2016 20:01:06 -0800 (PST) Date: Fri, 19 Feb 2016 23:00:50 -0500 From: Alexander Kabaev To: =?KOI8-R?Q?Jean-S'ebastien_P'edron?= Cc: freebsd-arch@FreeBSD.org Subject: Re: "initial-exec" TLS model and dlopen(3) Message-ID: <20160219230050.2586a92e@kan> In-Reply-To: <56C7A00A.3030802@FreeBSD.org> References: <56C7A00A.3030802@FreeBSD.org> X-Mailer: Claws Mail 3.13.1 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/RNG1uvgPVLqRYJjzX1A9/J8"; protocol="application/pgp-signature" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Feb 2016 04:01:08 -0000 --Sig_/RNG1uvgPVLqRYJjzX1A9/J8 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi, This is, generally speaking, illegal use of initial-exec, but since this appears to be the common practice Linux supports, so should we. I am working on a patch that will support this type of TLS abuse and I will send you one for testing once complete. =20 --=20 Alexander Kabaev --Sig_/RNG1uvgPVLqRYJjzX1A9/J8 Content-Type: application/pgp-signature Content-Description: Цифровая подпись OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWx+T0XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDNUY3RDk5NTk5QjY0MUUxM0M1MTU2OTEw NzEzMjI5OTkyNzkyRTdFAAoJEAcTIpmSeS5+7qkQAIXUzWs3rx5b9WqmtuX6sssm Hcg6XVJtqnBZJpY2oJlbFh8+qGC75eaKWxgTI2TNH0eIeFtQzfbHEJ+q5/HDxeFK mo/FEvYzbE7tKnByw9oUc2X2FDikZVua+APiFrW5xduB+rg0HOMEZ/k8wxf0J1TJ ch7cTeF39VdCkLlnNn1ONWBqACgjd1hT3NKABT6GXdYl3FkrlCcGHstV8WTJO48t F0NeT13R55Kr1BQq9vau3P8MKzzj3XI8cwO/g5GbStH7xfL2O8uyIJMeOK89gNR7 rJIN/ffxT4U5jyJstkbbUBzkQcZ72bhTzAd5QO558NfaTvQXE7eE7VWsdyoVKpS0 UTb9Q+pmTKNJF3mU4Ab1xRxmqP2PS6bs8zFdFU6WVZQl8FQRTvzGdthJzjLkX8NH gBXewfY692a1ZVk19Z7qyYymVfsS4nCrw/8wVg0Q2Tbp/yjgLe0lUA1vdycJMEha 2RqHloVB4JfPG7xe6zeidAb5bc7PoMqdKZSeKpJwVtPGkh4b6s+v4muI5JvyoYWA rs1xLF8T63fqTwK4EOWHY5rxIbO5G9riiDXccpNFwO0ifXJ4srCaDlzzxWjBuD82 mfKRJ/JK5GoZvGETvgQ5cF31LElhqhVSWDV+qzh6JJ5gLlzmAekZ37pa/OpyMrph ITyTl/LESaRMB9Y2JtgT =ueW6 -----END PGP SIGNATURE----- --Sig_/RNG1uvgPVLqRYJjzX1A9/J8--