From owner-freebsd-hackers@freebsd.org Sun Mar 26 04:13:05 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E62BD12AF6; Sun, 26 Mar 2017 04:13:05 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-6.mit.edu (dmz-mailsec-scanner-6.mit.edu [18.7.68.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D48AE1011; Sun, 26 Mar 2017 04:13:04 +0000 (UTC) (envelope-from kaduk@mit.edu) X-AuditID: 12074423-1a7ff70000005559-2f-58d73e99e698 Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 81.20.21849.99E37D85; Sun, 26 Mar 2017 00:07:53 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id v2Q47qTp019273; Sun, 26 Mar 2017 00:07:52 -0400 Received: from kduck.kaduk.org (24-107-191-124.dhcp.stls.mo.charter.com [24.107.191.124]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id v2Q47miA002608 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 26 Mar 2017 00:07:51 -0400 Date: Sat, 25 Mar 2017 23:07:47 -0500 From: Benjamin Kaduk To: freebsd-hackers@FreeBSD.org Cc: freebsd-current@FreeBSD.org Subject: Second call for 2017Q1 quarterly status reports Message-ID: <20170326040747.GV30306@kduck.kaduk.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.6.1 (2016-04-27) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrEIsWRmVeSWpSXmKPExsUixCmqrTvT7nqEwblDuha7rp1mt5jz5gOT xfbN/xgdmD1mfJrPEsAYxWWTkpqTWZZapG+XwJXRtPIFW8Fs7opDjyczNzAu5+xi5OSQEDCR WDCpgbWLkYtDSKCNSeLEkWMsEM5GRok/a84wQThXmST2/trIDNLCIqAqcfDlPSYQm01ATWL9 imtgcREBeYl9Te/ZQWxmIPvX1iYgm4NDWMBC4v6XBJAwL9C2bUc+M0PYghInZz5hgSjXkrjx 7yUTSDmzgLTE8n8cIGFRAWWJhhkPmCcw8s1C0jELSccshI4FjMyrGGVTcqt0cxMzc4pTk3WL kxPz8lKLdM30cjNL9FJTSjcxgoKO3UV5B+PLPu9DjAIcjEo8vA07r0UIsSaWFVfmHmKU5GBS EuWNOAsU4kvKT6nMSCzOiC8qzUktPsQowcGsJML7y+B6hBBvSmJlVWpRPkxKmoNFSZxXXKMx QkggPbEkNTs1tSC1CCYrw8GhJMHrbwvUKFiUmp5akZaZU4KQZuLgBBnOAzQ8BqSGt7ggMbc4 Mx0if4pRUUqc19kGKCEAksgozYPrBSUFiez9Na8YxYFeEeaNAGnnASYUuO5XQIOZgAbP3nAF ZHBJIkJKqoHRcU3cx77Gq7qx1xZ0Tk+en+/0oPu98eMlSpzR5s+fnAtYk2Q/c0enx/WzQdHX Ra58jXzI4BUpY7D2bPUB/8ajBo8mWV55IJ7Ldi6y9ZzGt/mH6yrP16udZG70Tds7Xf937n1N QYe5oRt2HTSsmOHxNPpsE3ueRtns6iyTg4Vz7RLMbqQ8vnFLiaU4I9FQi7moOBEATnAzR+UC AAA= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Mar 2017 04:13:05 -0000 Dear FreeBSD Community, The deadline for the next FreeBSD Quarterly Status update is April 7, 2017, for work done in January through March. Status report submissions do not need to be very long. They may be about anything happening in the FreeBSD project and community, and provide a great way to inform FreeBSD users and developers about work that is underway and completed. Submission of reports is not restricted to committers; anyone doing anything interesting and FreeBSD related can -- and should -- write one! The preferred and easiest submission method is to use the XML generator [1] with the results emailed to the status report team at monthly@FreeBSD.org . (Do be sure, though, to save the form output and not the form itself!) There is also an XML template [2] that can be filled out manually and attached if preferred. For the expected content and style, please study our guidelines on how to write a good status report [3]. You can also review previous issues [4][5] for ideas on the style and format. We look forward to seeing your 2016Q4 reports! Thanks, Ben (on behalf of monthly@) [1] https://www.FreeBSD.org/cgi/monthly.cgi [2] https://www.FreeBSD.org/news/status/report-sample.xml [3] https://www.FreeBSD.org/news/status/howto.html [4] https://www.FreeBSD.org/news/status/report-2016-07-2016-09.html [5] https://www.FreeBSD.org/news/status/report-2016-10-2016-12.html From owner-freebsd-hackers@freebsd.org Sun Mar 26 16:15:36 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3338AD1EEA0 for ; Sun, 26 Mar 2017 16:15:36 +0000 (UTC) (envelope-from pkubaj@anongoth.pl) Received: from mail.anongoth.pl (anongoth.pl [88.156.79.165]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anongoth.pl", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1B021CD8; Sun, 26 Mar 2017 16:15:35 +0000 (UTC) (envelope-from pkubaj@anongoth.pl) Received: from anongoth.pl (89-73-165-162.dynamic.chello.pl [89.73.165.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: pkubaj@anongoth.pl) by mail.anongoth.pl (Postfix) with ESMTPSA id C6DB72D88F; Sun, 26 Mar 2017 18:06:05 +0200 (CEST) Authentication-Results: mail.anongoth.pl; dmarc=none header.from=anongoth.pl DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=anongoth.pl; s=ANONGOTH; t=1490544366; bh=SINIqf1xvuECvsnnFhgiWAqUfe4WOrUuKrcnYUqEk5M=; h=Date:From:To:Subject:References:In-Reply-To; b=r5WswpVOaJz5nCCNHOeFkuua26qpaQn37xH8o6DGhDWU/jlOiaLtRdMXtkKfUsxsL rSKuLsx1Ra9Nqd8gkxyk9O0/BgJ7G9eDh+cnracJP3Yswffqs1mL6QMXIfOUQM8zbV k0rwtveT2f1yxxjOmQgFA5P3iCUUXemhQf33TzXAF7tuErPkWzaWtAI5j7YIj6A3DS RLEEZCEgi+2gu7e1wEJ/4eGLTF/9ILkZKGocMjuzaU5ysvLKSjdT9Toz8uCp6ThpZM n7Lyr3eTA55wCXnEqjuum7CCORiGNdou30ky1hNk9002g5mB1IdpsPyrFbgU5cJfdy KcqjWWcnCZytw== Date: Sun, 26 Mar 2017 18:06:05 +0200 From: Piotr Kubaj To: freebsd-hackers@freebsd.org, jhb@freebsd.org Subject: Re: Freeze during booting of ASUS F2A85-M motherboard with Coreboot Message-ID: <20170326160605.GA68209@DESKTOP1> References: <20170225224425.GA93825@DESKTOP1> <20170318190527.GA70683@ThinkPad-X200.local> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="C7zPtVaVf+AK4Oqc" Content-Disposition: inline In-Reply-To: <20170318190527.GA70683@ThinkPad-X200.local> User-Agent: Mutt/1.8.0 (2017-02-23) X-Mailman-Approved-At: Sun, 26 Mar 2017 20:46:24 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Mar 2017 16:15:36 -0000 --C7zPtVaVf+AK4Oqc Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable If anyone would like to use Coreboot with FreeBSD on F2A85-M, I guess I hav= e found out how to make it possible. 1. Console / VT still doesn't work after the bootloader, but it turns out X= 11 starts fine, although there are restrictions. If you need console, you w= ill need to use serial. It's something I can live, though. 2. To make X11 start you need to use sc(4) - sorry, no vt(4), it doesn't wo= rk. 3. You need to add to /usr/local/etc/X11/xorg.conf.d/driver-nvidia.conf: Section "Device" Identifier "Card0" Driver "nvidia" EndSection This will vary when you use a GPU from other GPU maker. 4. CPU overheats when it does something intensive, but I have observed it o= n Linux as well. --=20 _______________________________________=20 / When I kill, the only thing I feel is \ \ recoil. / ---------------------------------------=20 \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || --C7zPtVaVf+AK4Oqc Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEycyIeNkkgohzsoorelmbhSCDnJ0FAljX5u0ACgkQelmbhSCD nJ3CRg/7BIoGJyvXqbbbBxYbknnyMsAERWjBc9cnmSLJmHaFQYDBnGerepSVqk17 q9iKlBOTFObh9yJD8qVVzkk59Bf+0pHv8N/r4+xuJDiS8rturgGWTTP+X4BHlM62 XRf1mk7FKW4/YMnZQBI+0cVyuPHb8W/mbpVXybaXv89cdJB30V5v+fPOQFP5Qk0z LBZf9VwFpq5CaqpVQ1QkjnMtXVbgS+oXCz9C1mUpB6QK4ktmNdKjNTGXni2qpFVt rBmpIUi6BXF/pya7Za5sPGPizPkLsUReI4hNBeGNpA+18iheFkWk8x7WdYHDgYld ung8twT9/9GcY+f2R5XbCj5FbOpeW79M2MtSTO+yDppHh2lI+l/t4/JfCDWWkqDx Bl17+LoyzYSkFMxdmLdJP/9IDt0lJOBhSZZLLxsFlFOHh9tOOhLpijrrbn1rV1UX rwaCassmVK0CnHEa+ts+LMHwQsDKOB5jc/R74Brswa5rBhRaMPesg9hjs3wYC8y5 5FTM3XUovJAxXHt7z4gYD8124ziOlLf9ajr32DbPUyLm4pV+HxBtu6XEG9jmaQJ8 EC5SqpSozH7FQZrBPZZ1MD8ePFfCLAJUn1gnl/y1eRGTFsPEnaXUUtaogrc/EnHp NZPtjM/nrqi2uSM/zzaOHkyB7zyXJYDIevlOGPQ582PVPyD8zEc= =X1oL -----END PGP SIGNATURE----- --C7zPtVaVf+AK4Oqc-- From owner-freebsd-hackers@freebsd.org Mon Mar 27 11:47:13 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB05ED1F3E9 for ; Mon, 27 Mar 2017 11:47:13 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22c.google.com (mail-wr0-x22c.google.com [IPv6:2a00:1450:400c:c0c::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7033CB26 for ; Mon, 27 Mar 2017 11:47:13 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22c.google.com with SMTP id w11so39383049wrc.3 for ; Mon, 27 Mar 2017 04:47:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=0LZ/82lJaXoxPRR5WEbTM7EqRVO6GZcSn2G6xY6VXJI=; b=sDQPUQViMcop3pZaNx6KisbaU8u+o9/azmz13G+QrHNFSY1Khz3WhmydJ9rckv9ljc TMa0o6YUA5Y0gC7wc6UZ+yVjW4VmVJw6g07HXw7HYfSgZP6fqLHOPeeZy8rtyhcvekAQ FV8a8MNU72d9B08LBNH7bzj1umt9JUs8z42CSDRs+gHHeEDehfte+wf31Q1xp4VwiPax 0xRdzGxEU8RKfjEWmYcCQ0Qr9tzCv9FHXFUZdadvDitw5LTsiGpboR9fZHEe7HbSBV1t oacSVIReHOj6XMzYZSH5iWSbgDsmszgjJPJwFJnmT4Bwnpbnwvznnva3zbJXvMg9cNK4 g5eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=0LZ/82lJaXoxPRR5WEbTM7EqRVO6GZcSn2G6xY6VXJI=; b=Xtk0/R86CHkyFCMFU2qQb6w2DLVARPLc5B4XZ6t6cYw/irK3PUCG8D+NcUWTocpLPM A0Ln/tlEqW1ye0kGsXir40xDAJ2H/rSeYAkBliuyTav7XsXF0VcTrlFYT/6LkuNsP/zN Ejr7tQocwZJfcgxv1wuCKEVYScLRQfJavgF6Zq+m9JjSCzEKiXrotVUuEXtkuB3hYTSa rlj1jSzXE5c3Lo8k/p/QIPvr8hU59WGfwYMay4NMFQ9aMrc1IlI07ttXPxdyDstE4Ib5 VQGwY5dG1/0J6SD1IWpvgPRgBnO3vNrjq0JwAIheRs2zCRVM1yg7XS0sXVv2jHz7d5Sr YwIQ== X-Gm-Message-State: AFeK/H2qUPpa9SrsdfiqIyi399dheGcSU8ajCBfFSwL3e/724AfsAi1i6k2BbzezSRz6+cqi X-Received: by 10.223.164.83 with SMTP id e19mr2808903wra.154.1490615231061; Mon, 27 Mar 2017 04:47:11 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id y65sm376945wrb.50.2017.03.27.04.47.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 04:47:10 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <27e1a828-5cd9-0755-50ca-d7143e7df117@multiplay.co.uk> <20161206125919.GQ54029@kib.kiev.ua> <8b502580-4d2d-1e1f-9e05-61d46d5ac3b1@multiplay.co.uk> <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> Date: Mon, 27 Mar 2017 12:47:11 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170317124437.GR16105@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 11:47:14 -0000 OK now the similar but unrelated issue with signal stacks is solved I've moved back to the initial issue. I've made some progress with a reproduction case as detailed here: https://github.com/golang/go/issues/15658#issuecomment-288747812 In short it seems that having a running child, while the parent runs GC, is some how responsible for memory corruption in the parent. The reason I believe this is if I run the same GC in the parent after the child exits instead of while its running, I've been unable to reproduce the issue. As the memory segments are COW then the issue might be in VM subsystem. In order to confirm / deny this I was wondering if there was a way to force a full copy of all segments for the child instead of using the COW optimisation. Is this something that would be relatively easy to hack into the kernel, and if so pointers would be appreciated. Regards Steve From owner-freebsd-hackers@freebsd.org Mon Mar 27 13:27:32 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4DDBD1F045 for ; Mon, 27 Mar 2017 13:27:32 +0000 (UTC) (envelope-from alexander.tarasikov@gmail.com) Received: from mail-ot0-x235.google.com (mail-ot0-x235.google.com [IPv6:2607:f8b0:4003:c0f::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A9265FEE for ; Mon, 27 Mar 2017 13:27:32 +0000 (UTC) (envelope-from alexander.tarasikov@gmail.com) Received: by mail-ot0-x235.google.com with SMTP id t8so25670837otf.3 for ; Mon, 27 Mar 2017 06:27:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=dVpQjtVMTq7Kmteax2mxCfVQjnpZIMCVG9/YDfnVD8w=; b=aNR30YhsLgfVxWVu6KBFCAg/fmSa4W5iBLvfagJe3z7Do19bZDhbN8T8SIUbJpJQ2+ i7h4V+/USsqntZMdNsMn0BjOdT5tJyiUPTaP3qc9hpq7nIxp5MCwJEM93gYT9sC9ITXO S3JnNRupB1froAoh1rgQDMbnvuuUwtCkEQlYwKywo6TD/pBhTLCR4m+NAun2EPDeB3Xi CsfR3rZ4PdufAxPwM1ylixgbL4qxZCrQr5Uft1m4Fr5z0ypAXPr6j0Fv1q/43F5wajMH +oIXnQeLXjNATiF4T5x8GjNouJaCV7WTPd5LCXF+0fzxwYC9ogPKw8nX+089WeAS3GHU d7Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=dVpQjtVMTq7Kmteax2mxCfVQjnpZIMCVG9/YDfnVD8w=; b=AIBhzLRa2csfN419YDqIu3FaODng53bdFuoagUx9jXn5eOSbG4+5VRZJtkzEDpAOV7 Zaed9Pn+vOrSU2dAi+MIyxNXcX0/asRIFyBS1dLt4XVkVkYFOHujl56tHAnIkrmuFtiW uEPYGT31VpRzHQHz6Eget9QNVYrOFpAc4/VbrhQJZfi2spr2AACS45zxVYGq4UDUQREz wm4FFi1aMnFl7pwdPUw94tWN3D4mLccLQhxVeX33RolYzzrP2XWk30aJ5hDMgs08LXCa 40kPRxPSlgbPxnKqWT9FmBnPSI4epE6sTZBgRm4IpQ1sKKM2V9NnSbSYs81Dq4deiZPN ngNw== X-Gm-Message-State: AFeK/H3xNVxtjbvFwRqCQURZPtJVnoMPkJJ5/VQvygOfVUK8WGGnQlEeP468/LHIXc40n8gIzNJ/UTAPxtHsIA== X-Received: by 10.157.83.27 with SMTP id g27mr12481765oth.160.1490621252003; Mon, 27 Mar 2017 06:27:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.102.10 with HTTP; Mon, 27 Mar 2017 06:27:31 -0700 (PDT) In-Reply-To: <20170320131235.GB86500@zxy.spb.ru> References: <20170320131235.GB86500@zxy.spb.ru> From: Alexander Tarasikov Date: Mon, 27 Mar 2017 15:27:31 +0200 Message-ID: Subject: Re: jmalloc in shared memory To: Slawa Olhovchenkov Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 13:27:33 -0000 Hi Slawa, I'm not sure jemalloc is supposed to be used this way, but you should be able to achieve this functionality in two ways: 1. Editing jemalloc's allocator ("src/pages.c") to allocate in your region instead of calling mmap() 2. Hook mmap() and brk(), sbrk() with LD_PRELOAD and provide implementations that will allocate in the shared memory Hope this helps. On Mon, Mar 20, 2017 at 2:12 PM, Slawa Olhovchenkov wrote: > How I can use jmalloc in shared memory? > > I.e. parent process do mmap w/ MAP_ANON|MAP_SHARED, create jmalloc > "instance" in this memory and use jmalloc routines for memory management > in this region. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- Regards, Alexander From owner-freebsd-hackers@freebsd.org Mon Mar 27 13:53:42 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E0C7D1FB5D for ; Mon, 27 Mar 2017 13:53:42 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EABC76E7 for ; Mon, 27 Mar 2017 13:53:41 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1csV5R-000Nqk-GO; Mon, 27 Mar 2017 16:53:33 +0300 Date: Mon, 27 Mar 2017 16:53:33 +0300 From: Slawa Olhovchenkov To: Alexander Tarasikov Cc: freebsd-hackers@freebsd.org Subject: Re: jmalloc in shared memory Message-ID: <20170327135333.GD70430@zxy.spb.ru> References: <20170320131235.GB86500@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 13:53:42 -0000 On Mon, Mar 27, 2017 at 03:27:31PM +0200, Alexander Tarasikov wrote: > Hi Slawa, > I'm not sure jemalloc is supposed to be used this way, but you should > be able to achieve this functionality in two ways: > 1. Editing jemalloc's allocator ("src/pages.c") to allocate in your > region instead of calling mmap() > 2. Hook mmap() and brk(), sbrk() with LD_PRELOAD and provide > implementations that will allocate in the shared memory > > Hope this helps. I am don't need to redirect ALL allocations in the shared memory. I am need only do it for selected structures. For example, I am need create red-black tree and update it (in shared memory). For this, I am need create own memory management in this region or use existing memory management tool (for allocate, dealloacate and tracks chunks) worked for dedicated segment only. > On Mon, Mar 20, 2017 at 2:12 PM, Slawa Olhovchenkov wrote: > > How I can use jmalloc in shared memory? > > > > I.e. parent process do mmap w/ MAP_ANON|MAP_SHARED, create jmalloc > > "instance" in this memory and use jmalloc routines for memory management > > in this region. > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > > -- > Regards, Alexander From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:18:45 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB3D9D20CD1 for ; Mon, 27 Mar 2017 16:18:45 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B67D1D8; Mon, 27 Mar 2017 16:18:45 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2RGIXbt012419 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 27 Mar 2017 19:18:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2RGIXbt012419 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2RGIX4n012418; Mon, 27 Mar 2017 19:18:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 27 Mar 2017 19:18:33 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170327161833.GL43712@kib.kiev.ua> References: <20161206125919.GQ54029@kib.kiev.ua> <8b502580-4d2d-1e1f-9e05-61d46d5ac3b1@multiplay.co.uk> <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:18:45 -0000 On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: > OK now the similar but unrelated issue with signal stacks is solved I've > moved back to the initial issue. > > I've made some progress with a reproduction case as detailed here: > https://github.com/golang/go/issues/15658#issuecomment-288747812 > > In short it seems that having a running child, while the parent runs GC, > is some how responsible for memory corruption in the parent. > > The reason I believe this is if I run the same GC in the parent after > the child exits instead of while its running, I've been unable to > reproduce the issue. > > As the memory segments are COW then the issue might be in VM subsystem. Well, it might be, but it is a strange corruption mode to believe. > > In order to confirm / deny this I was wondering if there was a way to > force a full copy of all segments for the child instead of using the COW > optimisation. No, there is no. By design, copying only occurs on faults, when VM detects that the map entry needs copying. Doing the actual copy at fork time would require writing a lot of new code. Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? Another blind experiment to try is to comment out call to vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if it changes anything. What could be quite interesting is to look at the parent and possibly child address map after the error occured, using procstat -v. At least for parent, this should be relatively easy to set up, just make go runtime spin or pause on panic, instead of exiting, and then use procstat. > > Is this something that would be relatively easy to hack into the kernel, > and if so pointers would be appreciated. BTW, I looked some more at the go code, and I noted that runtimemmap() implementation looks very strange. It ignores %rflags.C bit to identify error, and instead callers of mmap() compare the return value with 4096, assuming Linux-style error reporting. This would certainly break if mmap(2) syscall returns ERESTART one day. From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:33:52 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23F38D202A3 for ; Mon, 27 Mar 2017 16:33:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6336ECE for ; Mon, 27 Mar 2017 16:33:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22d.google.com with SMTP id w11so49850673wrc.3 for ; Mon, 27 Mar 2017 09:33:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=CKBdDx+MF0i+Gx90tMikTVPxzLLyyOCYhaDw62Oa1fc=; b=06Q0KcFNh1MsX3NsAf5rdCJGx7xoHuR/kpSEdS9LT0P1oh1gVDyEwodAGgmHn7sCup mtzCLATNnX763WObPF7YWRvrnV/n5/9ElH0jDItCRLGcgXIiXsGaxp+x56ogUjKGogQ1 2Qo9iE7ZSLvDfReJh3ItVN8E66UiZRtstJR2BrwMynS+yPUIuFFuQ/JaLb0xdDPL3v/e hvYsw2eEL/1mZOEQiusxll1v2m9fLaViWpZraMsqRZeRYcB6GYTou8VF9LoZYlzEcWuq MqnEv2MkBTwT0tKMbc23Q4NC4GT78yeWBrAR3l+BifA9UXCHbTbt7QNnl89YVOiRUcYS kvFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=CKBdDx+MF0i+Gx90tMikTVPxzLLyyOCYhaDw62Oa1fc=; b=Lo8AfL9MVQOCioPneuX32mzQrarnC3e2tv2W6AdP9m9imYGDlhYYA0TZdKI4fqNVdA PgPC05byRi22fP8CGvIAW4ZH4GEE5bUV2NJ5/hOdE0XnDGN3Dv2KPq9UPPKRO8RuHgYj +z9/qNdHB4w2CIeIHqT+LdPg6WEDz0DH4ew8MYCucP1FYdshBYK70GMBEF0YCZ6Sduc9 ygS0AhTGYryJF0W8yvDQ+ugm7JLU4Lb4tKTgP3Q1lYgSy4ESsGayQCjasyBbbEoSYS4s JvV8dSMP4f9FHhCoLmCiCxTk50fTn1LZj6KsJQJhJujCnEdXZ4TywNhO8w6WJPukrOUq /5WA== X-Gm-Message-State: AFeK/H1LNRzj2PtvQJMa265Pe2GV1WvrNv/My5vqUZGEBBi5qamRk3/qrOeqA0u1NdpU8RcE X-Received: by 10.28.101.68 with SMTP id z65mr10433251wmb.102.1490632428750; Mon, 27 Mar 2017 09:33:48 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id b13sm110026wmf.6.2017.03.27.09.33.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 09:33:48 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206125919.GQ54029@kib.kiev.ua> <8b502580-4d2d-1e1f-9e05-61d46d5ac3b1@multiplay.co.uk> <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> Date: Mon, 27 Mar 2017 17:33:49 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327161833.GL43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:33:52 -0000 On 27/03/2017 17:18, Konstantin Belousov wrote: > On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >> OK now the similar but unrelated issue with signal stacks is solved I've >> moved back to the initial issue. >> >> I've made some progress with a reproduction case as detailed here: >> https://github.com/golang/go/issues/15658#issuecomment-288747812 >> >> In short it seems that having a running child, while the parent runs GC, >> is some how responsible for memory corruption in the parent. >> >> The reason I believe this is if I run the same GC in the parent after >> the child exits instead of while its running, I've been unable to >> reproduce the issue. >> >> As the memory segments are COW then the issue might be in VM subsystem. > Well, it might be, but it is a strange corruption mode to believe. Indeed, but would you agree the evidence seems to indicate that this may be the case, as otherwise I would have expected that running the GC after the child process has exited would have zero impact on the issue. > >> In order to confirm / deny this I was wondering if there was a way to >> force a full copy of all segments for the child instead of using the COW >> optimisation. > No, there is no. By design, copying only occurs on faults, when VM > detects that the map entry needs copying. Doing the actual copy at fork > time would require writing a lot of new code. I noticed in vm_map_copy_entry the following: /* * We don't want to make writeable wired pages copy-on-write. * Immediately copy these pages into the new map by simulating * page faults. The new pages are pageable. */ vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, fork_charge); I wondered if I could use vm_fault_copy_entry to force the copy on fork? > Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? Yes it does, I don't currently have i386 machine to test with, I'm assuming testing i386 on amd64 kernel, would likely not have any effect. > Another blind experiment to try is to comment out call to > vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if > it changes anything. I'll do that shortly. > What could be quite interesting is to look at the parent and possibly > child address map after the error occured, using procstat -v. At > least for parent, this should be relatively easy to set up, just make > go runtime spin or pause on panic, instead of exiting, and then use > procstat. I've been looking at the output from procstat -v I have seen the parent FLAGS ping ping between C--- and CN--, not sure if that's relevant e.g. procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 3 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 3 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 18 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 38 1 0 C--- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 1 0 C--- df 27099 0x800c1e000 0x800c6e000 rw- 18 18 1 0 C--- df 27099 0x800c6e000 0x800cae000 rw- 2 2 1 0 C--- df 27099 0x800cae000 0x800cee000 rw- 2 2 1 0 C--- df 27099 0x800cee000 0x800dae000 rw- 5 5 1 0 C--- df 27099 0x800dae000 0x800dee000 rw- 1 1 1 0 C--- df 27099 0x800dee000 0x800e2e000 rw- 1 1 1 0 C--- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 1 0 C--- df 27099 0x800e6e000 0x800eae000 rw- 1 1 1 0 C--- df 27099 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 37 0 ---- ph procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 38 2 0 CN-- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df 27099 0x800c1e000 0x800c6e000 rw- 18 18 2 0 CN-- df 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df 27099 0x800dee000 0x800e2e000 rw- 1 1 2 0 CN-- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 CN-D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 0 1 0 C--- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df 27099 0x800c1e000 0x800c6e000 rw- 18 0 1 0 C--- df 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df 27099 0x800dee000 0x800e2e000 rw- 1 0 1 0 C--- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 0 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph I'll definitely try capturing the output on fault, see what that looks like > >> Is this something that would be relatively easy to hack into the kernel, >> and if so pointers would be appreciated. > BTW, I looked some more at the go code, and I noted that > runtimemmap() implementation looks very strange. > It ignores %rflags.C bit to identify error, and instead callers > of mmap() compare the return value with 4096, assuming Linux-style > error reporting. This would certainly break if mmap(2) syscall > returns ERESTART one day. I'll look at this too, thanks for the heads up. Regards Steve From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:36:23 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F03FD204B9 for ; Mon, 27 Mar 2017 16:36:23 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F10EDB for ; Mon, 27 Mar 2017 16:36:23 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id 7CA6A5A9F15; Mon, 27 Mar 2017 16:26:37 +0000 (UTC) Date: Mon, 27 Mar 2017 16:26:37 +0000 From: Brooks Davis To: Slawa Olhovchenkov Cc: Alexander Tarasikov , freebsd-hackers@freebsd.org Subject: Re: jmalloc in shared memory Message-ID: <20170327162637.GB59667@spindle.one-eyed-alien.net> References: <20170320131235.GB86500@zxy.spb.ru> <20170327135333.GD70430@zxy.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="mP3DRpeJDSE+ciuQ" Content-Disposition: inline In-Reply-To: <20170327135333.GD70430@zxy.spb.ru> User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:36:23 -0000 --mP3DRpeJDSE+ciuQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 27, 2017 at 04:53:33PM +0300, Slawa Olhovchenkov wrote: > On Mon, Mar 27, 2017 at 03:27:31PM +0200, Alexander Tarasikov wrote: >=20 > > Hi Slawa, > > I'm not sure jemalloc is supposed to be used this way, but you should > > be able to achieve this functionality in two ways: > > 1. Editing jemalloc's allocator ("src/pages.c") to allocate in your > > region instead of calling mmap() > > 2. Hook mmap() and brk(), sbrk() with LD_PRELOAD and provide > > implementations that will allocate in the shared memory > >=20 > > Hope this helps. >=20 > I am don't need to redirect ALL allocations in the shared memory. > I am need only do it for selected structures. > For example, I am need create red-black tree and update it (in shared > memory). For this, I am need create own memory management in this > region or use existing memory management tool (for allocate, > dealloacate and tracks chunks) worked for dedicated segment only. JEMalloc is almost certainly overkill for this application. It sounds likely you're allocating fixed sized objects (or at least not too many of them). For that, a simple slab allocator should do the trick. Alterntively, rtld contains a simple pooled malloc that's not hard to adapt to random backing stores (libexec/rtld-elf/malloc.c). I've got version adapted for CHERI with pluggable storage backends at https://github.com/CTSRD-CHERI/cheribsd/tree/master/lib/libmalloc_simple. It wouldn't be too hard turn it back into portable C. -- Brooks --mP3DRpeJDSE+ciuQ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJY2T08AAoJEKzQXbSebgfAQJEIAIlsIw4RJmOAvYTQk4K5+rwS hSbCdySnQ46rlfHO5RKaVVklt4CF9Le4uxHYRUkPYttP7hzxebuulNN3nwK7CjEN GVlXX8RNmWfD5YeJTfpo93UPdArvooaMwGETtBUPWVdjxLEbWotb4RPsiRBXxA4Z F9JpLI/tcXnCWK4qaUR/Os1aZtZjfAGcavEvdmkZmGcu/cc++cw9X5zH3LSAaaUi tY3SLYws2ixptwzOhuliXV4voacOcDMvZN8W8a7d6Ug5J7F/pI0LJHt/5bGtx8J4 b59a+EnoGQRcou/yPt+v6zb0V4HKtTmpvdLEE7DsDJvsIT5wKVGjEaJl56RL8JE= =KqmF -----END PGP SIGNATURE----- --mP3DRpeJDSE+ciuQ-- From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:44:27 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43C7DD207ED for ; Mon, 27 Mar 2017 16:44:27 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1834913; Mon, 27 Mar 2017 16:44:26 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1csXkm-0000mC-4J; Mon, 27 Mar 2017 19:44:24 +0300 Date: Mon, 27 Mar 2017 19:44:24 +0300 From: Slawa Olhovchenkov To: Brooks Davis Cc: Alexander Tarasikov , freebsd-hackers@freebsd.org Subject: Re: jmalloc in shared memory Message-ID: <20170327164424.GE70430@zxy.spb.ru> References: <20170320131235.GB86500@zxy.spb.ru> <20170327135333.GD70430@zxy.spb.ru> <20170327162637.GB59667@spindle.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170327162637.GB59667@spindle.one-eyed-alien.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:44:27 -0000 On Mon, Mar 27, 2017 at 04:26:37PM +0000, Brooks Davis wrote: > On Mon, Mar 27, 2017 at 04:53:33PM +0300, Slawa Olhovchenkov wrote: > > On Mon, Mar 27, 2017 at 03:27:31PM +0200, Alexander Tarasikov wrote: > > > > > Hi Slawa, > > > I'm not sure jemalloc is supposed to be used this way, but you should > > > be able to achieve this functionality in two ways: > > > 1. Editing jemalloc's allocator ("src/pages.c") to allocate in your > > > region instead of calling mmap() > > > 2. Hook mmap() and brk(), sbrk() with LD_PRELOAD and provide > > > implementations that will allocate in the shared memory > > > > > > Hope this helps. > > > > I am don't need to redirect ALL allocations in the shared memory. > > I am need only do it for selected structures. > > For example, I am need create red-black tree and update it (in shared > > memory). For this, I am need create own memory management in this > > region or use existing memory management tool (for allocate, > > dealloacate and tracks chunks) worked for dedicated segment only. > > JEMalloc is almost certainly overkill for this application. It sounds > likely you're allocating fixed sized objects (or at least not too many > of them). For that, a simple slab allocator should do the trick. I am planed to allocate (and free) also strings. JEMalloc already present in libc. I am mean JEMalloc allow to create arena/some_other_stuff and point this stuff in call. > Alterntively, rtld contains a simple pooled malloc that's not hard > to adapt to random backing stores (libexec/rtld-elf/malloc.c). I've > got version adapted for CHERI with pluggable storage backends at > https://github.com/CTSRD-CHERI/cheribsd/tree/master/lib/libmalloc_simple. > It wouldn't be too hard turn it back into portable C. Thanks, I am discover this. From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:49:12 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CFC0D20931 for ; Mon, 27 Mar 2017 16:49:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4522AC73; Mon, 27 Mar 2017 16:49:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2RGn64x019079 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 27 Mar 2017 19:49:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2RGn64x019079 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2RGn5MM019078; Mon, 27 Mar 2017 19:49:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 27 Mar 2017 19:49:05 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170327164905.GN43712@kib.kiev.ua> References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:49:12 -0000 On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: > On 27/03/2017 17:18, Konstantin Belousov wrote: > > On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: > >> OK now the similar but unrelated issue with signal stacks is solved I've > >> moved back to the initial issue. > >> > >> I've made some progress with a reproduction case as detailed here: > >> https://github.com/golang/go/issues/15658#issuecomment-288747812 > >> > >> In short it seems that having a running child, while the parent runs GC, > >> is some how responsible for memory corruption in the parent. > >> > >> The reason I believe this is if I run the same GC in the parent after > >> the child exits instead of while its running, I've been unable to > >> reproduce the issue. > >> > >> As the memory segments are COW then the issue might be in VM subsystem. > > Well, it might be, but it is a strange corruption mode to believe. > Indeed, but would you agree the evidence seems to indicate that this may > be the case, as otherwise I would have expected that running the GC > after the child process has exited would have zero impact on the issue. > > > >> In order to confirm / deny this I was wondering if there was a way to > >> force a full copy of all segments for the child instead of using the COW > >> optimisation. > > No, there is no. By design, copying only occurs on faults, when VM > > detects that the map entry needs copying. Doing the actual copy at fork > > time would require writing a lot of new code. > I noticed in vm_map_copy_entry the following: > /* > * We don't want to make writeable wired pages > copy-on-write. > * Immediately copy these pages into the new map by > simulating > * page faults. The new pages are pageable. > */ > vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, > fork_charge); > > I wondered if I could use vm_fault_copy_entry to force the copy on fork? No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot page in not yet touched page, and the result is also wired. > > Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? > Yes it does, I don't currently have i386 machine to test with, I'm > assuming testing i386 on amd64 kernel, would likely not have any effect. Only if the bug is in kernel and not in the go runtime. I am still not convinced that the kernel is the culprit. > > Another blind experiment to try is to comment out call to > > vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if > > it changes anything. > I'll do that shortly. > > What could be quite interesting is to look at the parent and possibly > > child address map after the error occured, using procstat -v. At > > least for parent, this should be relatively easy to set up, just make > > go runtime spin or pause on panic, instead of exiting, and then use > > procstat. > I've been looking at the output from procstat -v I have seen the parent > FLAGS ping ping between C--- and CN--, not sure if that's relevant e.g. C means that the entry is COW, N means that COW was not yet applied after the last fork, i.e. there were no write access. I am interested in the procstat -v output after the failure. I understand that there should be no surprising data for normally executing code. > procstat -v 27099 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 27099 0x400000 0x70d000 r-x 309 635 3 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x70d000 0x94e000 r-- 270 635 3 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn > /root/golang/src/test5/test5 > 27099 0x985000 0x9a8000 rw- 18 18 1 0 C--- df > 27099 0x80094e000 0x800b4e000 rw- 38 38 1 0 C--- df > 27099 0x800b4e000 0x800c1e000 rw- 28 28 1 0 C--- df > 27099 0x800c1e000 0x800c6e000 rw- 18 18 1 0 C--- df > 27099 0x800c6e000 0x800cae000 rw- 2 2 1 0 C--- df > 27099 0x800cae000 0x800cee000 rw- 2 2 1 0 C--- df > 27099 0x800cee000 0x800dae000 rw- 5 5 1 0 C--- df > 27099 0x800dae000 0x800dee000 rw- 1 1 1 0 C--- df > 27099 0x800dee000 0x800e2e000 rw- 1 1 1 0 C--- df > 27099 0x800e2e000 0x800e6e000 rw- 1 1 1 0 C--- df > 27099 0x800e6e000 0x800eae000 rw- 1 1 1 0 C--- df > 27099 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df > 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df > 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df > 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df > 27099 0x7ffffffff000 0x800000000000 r-x 1 1 37 0 ---- ph > > procstat -v 27099 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn > /root/golang/src/test5/test5 > 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df > 27099 0x80094e000 0x800b4e000 rw- 38 38 2 0 CN-- df > 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df > 27099 0x800c1e000 0x800c6e000 rw- 18 18 2 0 CN-- df > 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df > 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df > 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df > 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df > 27099 0x800dee000 0x800e2e000 rw- 1 1 2 0 CN-- df > 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df > 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df > 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df > 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 CN-D df > 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph > > procstat -v 27099 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn > /root/golang/src/test5/test5 > 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn > /root/golang/src/test5/test5 > 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df > 27099 0x80094e000 0x800b4e000 rw- 38 0 1 0 C--- df > 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df > 27099 0x800c1e000 0x800c6e000 rw- 18 0 1 0 C--- df > 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df > 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df > 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df > 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df > 27099 0x800dee000 0x800e2e000 rw- 1 0 1 0 C--- df > 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df > 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df > 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 27099 0xc41fff8000 0xc420200000 rw- 255 0 1 0 C--- df > 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df > 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph > > I'll definitely try capturing the output on fault, see what that looks like > > > >> Is this something that would be relatively easy to hack into the kernel, > >> and if so pointers would be appreciated. > > BTW, I looked some more at the go code, and I noted that > > runtimemmap() implementation looks very strange. > > It ignores %rflags.C bit to identify error, and instead callers > > of mmap() compare the return value with 4096, assuming Linux-style > > error reporting. This would certainly break if mmap(2) syscall > > returns ERESTART one day. > I'll look at this too, thanks for the heads up. > > Regards > Steve From owner-freebsd-hackers@freebsd.org Mon Mar 27 17:54:53 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0EE8D2007F; Mon, 27 Mar 2017 17:54:53 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id A7E3017D; Mon, 27 Mar 2017 17:54:53 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f] (unknown [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id AA70E1658; Mon, 27 Mar 2017 17:54:52 +0000 (UTC) To: "freebsd-hackers@freebsd.org" Cc: freebsd-security@freebsd.org From: Eric McCorkle Subject: Proposal for a design for signed kernel/modules/etc Message-ID: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> Date: Mon, 27 Mar 2017 13:54:44 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="5tJFchSClLXPlLXBTeF3xliHxegxS2CGS" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 17:54:54 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --5tJFchSClLXPlLXBTeF3xliHxegxS2CGS Content-Type: multipart/mixed; boundary="VfKDQGqN42otawkWgJ1TC8hPFp8JNnhaw"; protected-headers="v1" From: Eric McCorkle To: "freebsd-hackers@freebsd.org" Cc: freebsd-security@freebsd.org Message-ID: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> Subject: Proposal for a design for signed kernel/modules/etc --VfKDQGqN42otawkWgJ1TC8hPFp8JNnhaw Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello everyone, The following is a design proposal for signed kernel and kernel module loading, both at boot- and runtime (with the possibility open for signed executables and libraries if someone wanted to go that route). I'm interested in feedback on the idea before I start actually writing code for it. =3D=3D Goals =3D=3D 1) Be able to check for a correct cryptographic signature for any kernel or modules loaded at boot time for some platforms (EFI at a minimum). 2) Be able to check for a correct cryptographic signature for any kernel module loaded during normal operations (whether or not to do this could be controlled by a sysctl, securelevel, or some similar mechanism) 3) Work with what's in base already and minimize new additions (ideally, just a small utility to sign executables) 4) Minimize administrative overhead and ideally, require no changes at all to maintain signed kernel/modules 5) Have a clear path for supporting signed executables/libraries (I'm not planning on pursuing this, but it's worth considering that someone might want to) 6) The design *MUST* support the case where a system builds locally and uses its own key(s) for signing kernels and modules (and anything else) and *MUST* allow the administrator complete control over which key(s) are valid for a given system (ie. no "master keys" controlled by central organizations) 7) The design must allow for the adoption of new ciphers (there is an inevitable shift to post-quantum ciphers coming in the near future) =3D=3D Non-Goals =3D=3D * Hardware/firmware-based attacks are considered out-of-scope (there is no viable method for defending against them at the OS level) * Boot platforms that don't provide their own signature-checking framework up to loader/kernel can't be properly secured, and are considered out-of-scope * Boot platforms that impose size restrictions prohibiting incorporation of RSA and ED25519 crypto code (ex. i386 BIOS) are considered out-of-scop= e * GRUB support is desirable, however it is not necessary to support GRUB out-of-the-box (meaning a design requiring reasonable modifications to GRUB is acceptable). * I am not aiming to support signed executables/libraries now, only to avoid shutting the door on them. =3D=3D Existing Solution(s) =3D=3D EFI has a decent design with regard to key management (platform key, which signs system keys, which sign the actual loader); however, its cipher suite is sorely lacking (many broken hashes and weak ciphers, RSA 2048 being the "strongest", no ECC). It also only works with the COFF format, and is only available at boot time. However, it does provide a chain of custody up to loader (to the extent that anyone trusts closed-source firmware blobs, SHA1, and 512-2048 bit RSA keys...) Many implementations also have master keys "baked in" that would allow anything signed by random third parties (Microsoft) to boot regardless of local configurations, or they don't provide any sort of control over (or even access to) the keys at all. EFI obviously isn't viable beyond boot time, and misses most of the goals even there. Its key management hierarchy is an overall good design, however. GRUB currently supports signature checking. It can be configured to require signatures for any of its own modules as well as any kernel or modules that it loads. These signatures are stored *outside* the executable/library, in a file with an added .sig extension. The format is that of an external signature for the entire ELF file as produced by the gnupg program. Linux (I believe) also supports signature checking for modules using the same convention. While functional, this design doesn't meet the goals I outlined: * It relies on the gnupg framework, which is not part of FreeBSD base, and adding it would be a chore (and would end up duplicating a lot of functionality provided by OpenSSL) * It stores the signature separate from the file, which leads to x2 the number of files, would require modifying existing scripts, and complicates administrative tasks. It also leads to failure modes like stale signatures. * There are potential legitimate modifications to non-code parts of an ELF file (such as the .comment section or other similar sections) that would require re-signing the entire file. * The previous two problems really start to look bad when you consider signed executables and libraries, possibly with third-party build/install scripts... * Finally, the gnupg signature format doesn't actually seem to be documented anywhere, or at least not anywhere that doesn't require a lot of digging... An alternate solution, which I believe is used in some places is to wrap the entire executable in a PGP envelope-like format. This solves the issue of an external signature file, but would require extensive modification to the ELF parsing code, let alone the binutils programs that read/modify ELF files. This solution also isn't backwards-compatible at all. Old loaders/kernels will choke on the signed libraries. =3D=3D Proposal=3D=3D My proposal is to store cryptographic signatures within the ELF files themselves in a non-loadable section (similar to the .comment section). As background, the ELF file format has a number of different section types, only some of which comprise the program/library/module's runtime state. The ELF specification and tools provide some "standard" sections with defined meanings, but nothing stops anyone from adding their own sections. The ELF file format is quite flexible, and it is not difficult to add custom metadata to an ELF file. [0] In this proposal, cryptographic signatures would be stored in a =2Esignature (or .sig) section. This section would contain an array of signature constructs: one for each loadable segment in the ELF file. Signatures are computed for the contents of the segment's file data (ie. the data from p_offset to p_filesz, for the corresponding program header entry) along with all data from its program header entry except for p_offset and p_filesz. This scheme allows the actual data to be moved around in the file, so long as it (or the relevant program header data) isn't modified. The exact format of this data can be discussed, but a design where the signature array corresponds to the program header array seems quite reasonable. The format of the signatures themselves should be something from a well-defined standard, reasonably extensible, and supported by tools in base. =3D=3D Summary of Changes =3D=3D The following changes would be required: 1) Add a userland utility for signing ELF files (call it "signelf"). This would be a pretty straightforward application of OpenSSL and libelf.= 2) Modify ELF-parsing code in loader and kernel to check signatures and indicate whether a given file had good signatures for all of its loadable segments. 3) Have loader/kernel issue warnings or reject kernels/modules with incomplete/incorrect/no signatures 4) Decide how to go about building public key data into loader/kernel or how to register keys with the kernel (it is probably OK to implement a "bake it in" solution first, then figure out dynamic registration of keys as a follow-up; somebody out there is sure to want just the "bake it in" solution with no dynamic registration for security reasons, and we need it for loader anyway). 5) Submit a patch against GRUB to support the ELF metadata method in addition to their existing method. The most involved part of this is adding the public-key crypto code into loader and the kernel. My recommendation for this is to grab the RSA and ED25519 code from NaCL. It's compact, self-contained, written by crypto people with a good handle on the systems side of things (DJB's group), and licensed under a BSD-compatible license. Also, the loader/kernel side code only needs signature-checking, not full public-key functionality. =3D=3D Rationale =3D=3D The ELF metadata approach eliminates all of the disadvantages of the GRUB external signatures method, while maintaining compatibility with existing systems. Older systems will simply ignore the .signature metadata section and function normally and from a sysadmin standpoint, signed executables/libraries are just slightly larger versions of the unsigned variants. Moreover, ELF metadata that isn't part of the executable sections can be freely modified, and signed ELF files can be re-signed. Having a separate signature for each segment in the program header table is slightly more complicated than the simplest solution of having one signature for all program header sections. However, this approach provides more flexibility going forward. It also accounts for the fact that we might not want to sign all portions of the file. Finally, as designed, it allows the file to be modified freely as long as the runtime behavior isn't affected. There is a rather simple design possibility if anyone wanted to go the signed executable/library route: have an mmap variant with an additional parameter pointing to the signature would lead to a very simple modification of the userland dlopen functionality. Normal mmap would just become a wrapper around the secure variant, which passes in NULL for the signature (alternatively, you could pass in a default key built into the local libc, or something similar). =3D=3D Conclusion =3D=3D This seems like a good point in the design space: it doesn't break anything, it doesn't require massive changes or rearchitecting of anything, it provides everything I want to provide now, and it leaves the door open to things people might want to do in the future. Please provide feedback, comments, and suggestions. [0]: There actually is at least one example of something like this of which I'm aware. The Intel C Compiler (icc, "proton" by Intel internal naming) has an interprocedural optimization mode which produces .o files containing the compiler's intermediate representation in a special section as well as object code in the usual sections (incidentally, in the distant past, icc would actually produce separate .o and .il files; this was later changed to the ELF metadata solution, for the very reason that it complicated build scripts quite a bit). This allows "normal" compilers and compilation modes to use the object code, while icc uses the intermediate representation. --VfKDQGqN42otawkWgJ1TC8hPFp8JNnhaw-- --5tJFchSClLXPlLXBTeF3xliHxegxS2CGS Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEzzhiNNveVG6nWjcH1w0wQIFco2cFAljZUeQACgkQ1w0wQIFc o2cyhxAA2HlLeBeH5yEwzJyAq0oAkL0YVkFmrFUQpWh/jUWPvpWPmiWP/5m/lKnW ET1wttA2/2rIKa1N/G/Srm2NOSt38YHb4rTAs3fvcUi62mg4t7Lq1HDD1S9vLBtD EUe1agw9tHtGREmwVmkUaAKuprgNyDAhm0mbgjJpIk1KQpNHUojREIO8fzZUVzow KVtdRfmVJZOKFjk9Nl3Z7dDQoyWtQxnbP1Yss3jKXpgHwOfqgmFyqqGf0FtJ9G+Y IW7rb8wfMC8vTkblYFjR10jzpgd7wQNGa+bHb0qWQU5pdJA8mlLKnGVfUDxM2W8K 4eTNpKLHAi1avTz9oZ3tzigSgBBcf6s9kkfu50+lKpf8Cke/mH24sCGpLfCqb1Tu spcg0FbouvUhVMXT2zkZ9YgfTRyFcTcw+OIjCeWLr4vexIyMkdVwAPsZ7BK6E/bi odg9Pmm4Hy3SilSw8CIHGyUfx5UVZjUM4Ep6Y3jRoQ7CKiHC/lZK918R01qmJ5sR oDTyQnWo09Pf+w9/i6PP6Qo4313MJc/fk6QkWccbeWgaug9zVMw52nIS1d7i/lCw rS6Shq7pywEreAcc2FnMHXgafYJFVp+gPsNSTFXLCUjJS1ZbZUYGYA4XVkfnfBFf 5bCjT8R4amdp/OU5cOEEiMjGXqDnVjTX4XudFMEuHgZlC2CjBkU= =y53H -----END PGP SIGNATURE----- --5tJFchSClLXPlLXBTeF3xliHxegxS2CGS-- From owner-freebsd-hackers@freebsd.org Mon Mar 27 18:17:22 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BEFCD20821 for ; Mon, 27 Mar 2017 18:17:22 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6EE31B8 for ; Mon, 27 Mar 2017 18:17:21 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x236.google.com with SMTP id u1so69296601wra.2 for ; Mon, 27 Mar 2017 11:17:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=sW8cshDDrqP4lpAHEObHr1cyJ0HthpiTSBcDc+I2yLA=; b=jdp8J7Hkg3aVoJBIscb6HoVIE8Qm1FfZdDS5gglglfI0Pnc7BZLkeE7o59V6ehgx88 C7zy35sxyOpqXZzZAAPupssPVdzQfuCWchA8R4nEwLIIuMhCnRzR8FP8n7BFsARMafLv 7EkRVOQk1+dGCtyqzjR3csi7abfU6yJ/ifGBBN6u+c2iAcyMtMkh7wARntGqbU/TsU4V Kz+T56LcWhGa8h5qVpQz4Szygg/D6hMbIh6Pz4fV7BeQqVzRt9IgV6O1Jhm6mi5b7Pfr 28JjjEnOTqOxEQNmS50Cy9sSSH7OwQ5EgPOQJXkBjB82Fy99RdXBxdDCfNIm5jmm2mfp xE6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=sW8cshDDrqP4lpAHEObHr1cyJ0HthpiTSBcDc+I2yLA=; b=VdKmI32hvG3Ag5uHt6SiF96xcEvCymTyr4VYGakjFa94dHTFGVKFHFXBl2j9wQiW+a TyOxdcSa12DPD5DodE0Tyx6O8x3ouoKOEerUaC5whNoGCiLnM3UMxRKTsRwKsZoJKOKr yxfYgqLn4Lq9CfHt3n9gwy+iqrvbOw+ovzgi0FDHo84s8hi+9Xbl3qTZBIkElBFcUe6K iIj+iN5qryerSPK2a5KKvacJ5nypTYw+RDNXNf7yZQM9X1F6JSKvLCEtGJIZ9QPHYNDM Z6GfEZ2yEFn8HG4ybsHkjtGqRer8PYnV12qWXbf4OW37bxfvnkfKpAUCmk85JAxxyTLS M2Jg== X-Gm-Message-State: AFeK/H2yCeVNOJgatPp42Ai2IQwzdL3sLSC6yzmAvpChlP9AdIqm2hu2T4/SScSClUWsJlyG X-Received: by 10.28.55.3 with SMTP id e3mr11078771wma.141.1490638640155; Mon, 27 Mar 2017 11:17:20 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id o22sm1683234wro.13.2017.03.27.11.17.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 11:17:19 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: Date: Mon, 27 Mar 2017 19:17:20 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327164905.GN43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 18:17:22 -0000 On 27/03/2017 17:49, Konstantin Belousov wrote: > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: >> On 27/03/2017 17:18, Konstantin Belousov wrote: >>> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >>>> OK now the similar but unrelated issue with signal stacks is solved I've >>>> moved back to the initial issue. >>>> >>>> I've made some progress with a reproduction case as detailed here: >>>> https://github.com/golang/go/issues/15658#issuecomment-288747812 >>>> >>>> In short it seems that having a running child, while the parent runs GC, >>>> is some how responsible for memory corruption in the parent. >>>> >>>> The reason I believe this is if I run the same GC in the parent after >>>> the child exits instead of while its running, I've been unable to >>>> reproduce the issue. >>>> >>>> As the memory segments are COW then the issue might be in VM subsystem. >>> Well, it might be, but it is a strange corruption mode to believe. >> Indeed, but would you agree the evidence seems to indicate that this may >> be the case, as otherwise I would have expected that running the GC >> after the child process has exited would have zero impact on the issue. >>>> In order to confirm / deny this I was wondering if there was a way to >>>> force a full copy of all segments for the child instead of using the COW >>>> optimisation. >>> No, there is no. By design, copying only occurs on faults, when VM >>> detects that the map entry needs copying. Doing the actual copy at fork >>> time would require writing a lot of new code. >> I noticed in vm_map_copy_entry the following: >> /* >> * We don't want to make writeable wired pages >> copy-on-write. >> * Immediately copy these pages into the new map by >> simulating >> * page faults. The new pages are pageable. >> */ >> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, >> fork_charge); >> >> I wondered if I could use vm_fault_copy_entry to force the copy on fork? > No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot > page in not yet touched page, and the result is also wired. > >>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? >> Yes it does, I don't currently have i386 machine to test with, I'm >> assuming testing i386 on amd64 kernel, would likely not have any effect. > Only if the bug is in kernel and not in the go runtime. I am still not > convinced that the kernel is the culprit. > >>> Another blind experiment to try is to comment out call to >>> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if >>> it changes anything. >> I'll do that shortly. >>> What could be quite interesting is to look at the parent and possibly >>> child address map after the error occured, using procstat -v. At >>> least for parent, this should be relatively easy to set up, just make >>> go runtime spin or pause on panic, instead of exiting, and then use >>> procstat. >> I've been looking at the output from procstat -v I have seen the parent >> FLAGS ping ping between C--- and CN--, not sure if that's relevant e.g. > C means that the entry is COW, N means that COW was not yet applied after > the last fork, i.e. there were no write access. > > I am interested in the procstat -v output after the failure. I understand > that there should be no surprising data for normally executing code. Here's the output after its logged the errors. procstat -v 36159 PID START END PRT RES PRES REF SHD FLAG TP PATH 36159 0x400000 0x70d000 r-x 308 588 3 1 CN-- vn /root/golang/src/test5/test5 36159 0x70d000 0x94e000 r-- 248 588 3 1 CN-- vn /root/golang/src/test5/test5 36159 0x94e000 0x985000 rw- 31 0 1 0 C--- vn /root/golang/src/test5/test5 36159 0x985000 0x9a8000 rw- 18 18 1 0 C--- df 36159 0x80094e000 0x8009ee000 rw- 35 35 1 0 C--- df 36159 0x8009ee000 0x800abe000 rw- 28 28 1 0 C--- df 36159 0x800abe000 0x800ace000 rw- 16 16 1 0 C--- df 36159 0x800ace000 0x800b0e000 rw- 2 2 1 0 CN-- df 36159 0x800b0e000 0x800b4e000 rw- 2 2 1 0 C--- df 36159 0x800b4e000 0x800b8e000 rw- 2 2 1 0 C--- df 36159 0x800b8e000 0x800c0e000 rw- 3 3 1 0 C--- df 36159 0x800c0e000 0x800c4e000 rw- 2 2 1 0 C--- df 36159 0x800c4e000 0x800c8e000 rw- 1 1 1 0 CN-- df 36159 0x800c8e000 0x800cce000 rw- 1 1 1 0 C--- df 36159 0x800cce000 0x800d0e000 rw- 1 1 1 0 CN-- df 36159 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df 36159 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df 36159 0xc41fff8000 0xc420200000 rw- 258 258 1 0 C--- df 36159 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 36159 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph I've left it running in case you want any more info. Regards Steve From owner-freebsd-hackers@freebsd.org Mon Mar 27 18:37:38 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54237D2072F for ; Mon, 27 Mar 2017 18:37:38 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x229.google.com (mail-qk0-x229.google.com [IPv6:2607:f8b0:400d:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B12D947 for ; Mon, 27 Mar 2017 18:37:38 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x229.google.com with SMTP id p22so46619175qka.3 for ; Mon, 27 Mar 2017 11:37:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=RbKMRDO2TFZMhcGD7fNKwTsEG+dL/ZG80nITzzt9IH8=; b=HnmSZAA9WKbl7NfLByBOF5o30xVxW+W0m+VaKM76TNmfI2rFlvsKU8DWcnyeiX4GOL TY2MIoDJhfvJ8sD99yLSZoOHAX+kQn3tURSZlaxoVAcLtRIP74aeT+JhbvyQtAPEcctn WI/nZn5lMXWBciaqaOINq5v/J9gjMtMSsC9xXGZd9tN2zOVc22SEQ8uRo3ze4f7aPBuN ZOzMG+b7dZvFR52VeSanOragLS6ml3QmvUHt+Aofplp8t+6ixjxCRgduMWhUWxq0eA4O n5FkmOu3Muq3e6HJO2L0M1NzXo28+NSXQTrMtJxqLiRPks0NCkHhzsZ60aHR+jZEdiA6 cazQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=RbKMRDO2TFZMhcGD7fNKwTsEG+dL/ZG80nITzzt9IH8=; b=CoLBboNITgG0ayVcOaniL/g0fJetFa+HbOO5LQGen86i3ifXobugIu++mXxW4YvB3C RzfJSdmqqdgZUjsNR8Xd1axCoqh/u7LCWN1Wc8lRPLc4Ptfb2F+/d3URXk/UgUw+shd0 hxyjmn/b8ytV0rfniHixWBTIFxqPakTbud+EyhxvzzdkNQK2Qep6Wvqxi207LIiHJjQK wCzdy+HawPlzOr9CbGLf5hiycOw/F1WZDNxxn9cOVOxM/VvQXFgRpdkHtKD+gpBEGWum ymwPYA7QohvpXIqpnumk5MB7XvFk0SoAF8nHcS98C/s/sV9ra56VwAe79aSQXNeA1O7o VFuQ== X-Gm-Message-State: AFeK/H3fP7H2bPRRooY9PdTkq2HQ+H5O6UH0Gy9JfZqRE0ab2DEzQpfYprfVGzHAt1zo646A X-Received: by 10.55.134.5 with SMTP id i5mr19528502qkd.29.1490639856888; Mon, 27 Mar 2017 11:37:36 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.66]) by smtp.gmail.com with ESMTPSA id d4sm920000qkf.30.2017.03.27.11.37.36 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Mar 2017 11:37:36 -0700 (PDT) Date: Mon, 27 Mar 2017 14:37:35 -0400 From: Shawn Webb To: Eric McCorkle Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Subject: Re: Proposal for a design for signed kernel/modules/etc Message-ID: <20170327183735.uokjhjaafkawc2id@mutt-hbsd> References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="2iuyerr5quyn3eol" Content-Disposition: inline In-Reply-To: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> X-Operating-System: FreeBSD mutt-hbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20170206 (1.7.2) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 18:37:38 -0000 --2iuyerr5quyn3eol Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hey Eric, Thank you for writing this! ELF binary signing has been on my ever-growing list of things to research and develop. If you'd like help, please let me know. I have a few comments, which I've made inline. On Mon, Mar 27, 2017 at 01:54:44PM -0400, Eric McCorkle wrote: > Hello everyone, >=20 > The following is a design proposal for signed kernel and kernel module > loading, both at boot- and runtime (with the possibility open for signed > executables and libraries if someone wanted to go that route). I'm > interested in feedback on the idea before I start actually writing code > for it. >=20 > =3D=3D Goals =3D=3D >=20 > 1) Be able to check for a correct cryptographic signature for any kernel > or modules loaded at boot time for some platforms (EFI at a minimum). >=20 > 2) Be able to check for a correct cryptographic signature for any kernel > module loaded during normal operations (whether or not to do this could > be controlled by a sysctl, securelevel, or some similar mechanism) >=20 > 3) Work with what's in base already and minimize new additions (ideally, > just a small utility to sign executables) >=20 > 4) Minimize administrative overhead and ideally, require no changes at > all to maintain signed kernel/modules >=20 > 5) Have a clear path for supporting signed executables/libraries (I'm > not planning on pursuing this, but it's worth considering that someone > might want to) >=20 > 6) The design *MUST* support the case where a system builds locally and > uses its own key(s) for signing kernels and modules (and anything else) > and *MUST* allow the administrator complete control over which key(s) > are valid for a given system (ie. no "master keys" controlled by central > organizations) >=20 > 7) The design must allow for the adoption of new ciphers (there is an > inevitable shift to post-quantum ciphers coming in the near future) As git has shown, having a modular/configurable crypto interface is the best route. Right now, git is stuck using SHA1 because they didn't support users being able to choose which hashing algorithm to use. >=20 > =3D=3D Non-Goals =3D=3D >=20 > * Hardware/firmware-based attacks are considered out-of-scope (there is > no viable method for defending against them at the OS level) >=20 > * Boot platforms that don't provide their own signature-checking > framework up to loader/kernel can't be properly secured, and are > considered out-of-scope >=20 > * Boot platforms that impose size restrictions prohibiting incorporation > of RSA and ED25519 crypto code (ex. i386 BIOS) are considered out-of-scope >=20 > * GRUB support is desirable, however it is not necessary to support GRUB > out-of-the-box (meaning a design requiring reasonable modifications to > GRUB is acceptable). >=20 > * I am not aiming to support signed executables/libraries now, only to > avoid shutting the door on them. Since FreeBSD uses ELF for the kernel and its modules, it should be rather straightforward to apply the same work towards userland binaries and shared libraries. >=20 > =3D=3D Existing Solution(s) =3D=3D >=20 > EFI has a decent design with regard to key management (platform key, > which signs system keys, which sign the actual loader); however, its > cipher suite is sorely lacking (many broken hashes and weak ciphers, RSA > 2048 being the "strongest", no ECC). It also only works with the COFF > format, and is only available at boot time. However, it does provide a > chain of custody up to loader (to the extent that anyone trusts > closed-source firmware blobs, SHA1, and 512-2048 bit RSA keys...) Many > implementations also have master keys "baked in" that would allow > anything signed by random third parties (Microsoft) to boot regardless > of local configurations, or they don't provide any sort of control over > (or even access to) the keys at all. >=20 > EFI obviously isn't viable beyond boot time, and misses most of the > goals even there. Its key management hierarchy is an overall good > design, however. >=20 > GRUB currently supports signature checking. It can be configured to > require signatures for any of its own modules as well as any kernel or > modules that it loads. These signatures are stored *outside* the > executable/library, in a file with an added .sig extension. The format > is that of an external signature for the entire ELF file as produced by > the gnupg program. >=20 > Linux (I believe) also supports signature checking for modules using the > same convention. >=20 > While functional, this design doesn't meet the goals I outlined: >=20 > * It relies on the gnupg framework, which is not part of FreeBSD base, > and adding it would be a chore (and would end up duplicating a lot of > functionality provided by OpenSSL) >=20 > * It stores the signature separate from the file, which leads to x2 the > number of files, would require modifying existing scripts, and > complicates administrative tasks. It also leads to failure modes like > stale signatures. >=20 > * There are potential legitimate modifications to non-code parts of an > ELF file (such as the .comment section or other similar sections) that > would require re-signing the entire file. >=20 > * The previous two problems really start to look bad when you consider > signed executables and libraries, possibly with third-party > build/install scripts... >=20 > * Finally, the gnupg signature format doesn't actually seem to be > documented anywhere, or at least not anywhere that doesn't require a lot > of digging... >=20 >=20 > An alternate solution, which I believe is used in some places is to wrap > the entire executable in a PGP envelope-like format. This solves the > issue of an external signature file, but would require extensive > modification to the ELF parsing code, let alone the binutils programs > that read/modify ELF files. This solution also isn't > backwards-compatible at all. Old loaders/kernels will choke on the > signed libraries. Whatever is chosen, it should be fully-functional with only the utilities base provides. >=20 > =3D=3D Proposal=3D=3D >=20 > My proposal is to store cryptographic signatures within the ELF files > themselves in a non-loadable section (similar to the .comment section). >=20 > As background, the ELF file format has a number of different section > types, only some of which comprise the program/library/module's runtime > state. The ELF specification and tools provide some "standard" sections > with defined meanings, but nothing stops anyone from adding their own > sections. The ELF file format is quite flexible, and it is not > difficult to add custom metadata to an ELF file. [0] >=20 > In this proposal, cryptographic signatures would be stored in a > .signature (or .sig) section. This section would contain an array of > signature constructs: one for each loadable segment in the ELF file. > Signatures are computed for the contents of the segment's file data (ie. > the data from p_offset to p_filesz, for the corresponding program header > entry) along with all data from its program header entry except for > p_offset and p_filesz. This scheme allows the actual data to be moved > around in the file, so long as it (or the relevant program header data) > isn't modified. You might want to take a look at Microsoft's Authenticode. Microsoft made some mistakes early on that allowed attackers to easily trojan signed binaries. Your proposal up to this point makes those same mistakes. It's been a few years since I researched Authenticode, so I don't have any links or documentation handy. The conclusion Microsoft came to is that the file as a whole must be signed, including offset metadata. Essentially, you'd determine how large the .sig section needs to be ahead of time, create it and fill it with zeros, then sign the whole file, stuffing the signature in the zeroed .sig section. Same concept as calculating checksums of ICMP packets. This prevents attackers from modifying critical pieces of metadata, pointing them to maliciuos payloads. It also prevents attackers from appending malicious data to the end of a loadable segment (something Authenticode suffered from early on). >=20 > The exact format of this data can be discussed, but a design where the > signature array corresponds to the program header array seems quite > reasonable. The format of the signatures themselves should be something > from a well-defined standard, reasonably extensible, and supported by > tools in base. >=20 > =3D=3D Summary of Changes =3D=3D >=20 > The following changes would be required: >=20 > 1) Add a userland utility for signing ELF files (call it "signelf"). > This would be a pretty straightforward application of OpenSSL and libelf. >=20 > 2) Modify ELF-parsing code in loader and kernel to check signatures and > indicate whether a given file had good signatures for all of its > loadable segments. >=20 > 3) Have loader/kernel issue warnings or reject kernels/modules with > incomplete/incorrect/no signatures >=20 > 4) Decide how to go about building public key data into loader/kernel or > how to register keys with the kernel (it is probably OK to implement a > "bake it in" solution first, then figure out dynamic registration of > keys as a follow-up; somebody out there is sure to want just the "bake > it in" solution with no dynamic registration for security reasons, and > we need it for loader anyway). >=20 > 5) Submit a patch against GRUB to support the ELF metadata method in > addition to their existing method. >=20 > The most involved part of this is adding the public-key crypto code into > loader and the kernel. My recommendation for this is to grab the RSA > and ED25519 code from NaCL. It's compact, self-contained, written by > crypto people with a good handle on the systems side of things (DJB's > group), and licensed under a BSD-compatible license. Also, the > loader/kernel side code only needs signature-checking, not full > public-key functionality. >=20 > =3D=3D Rationale =3D=3D >=20 > The ELF metadata approach eliminates all of the disadvantages of the > GRUB external signatures method, while maintaining compatibility with > existing systems. Older systems will simply ignore the .signature > metadata section and function normally and from a sysadmin standpoint, > signed executables/libraries are just slightly larger versions of the > unsigned variants. Moreover, ELF metadata that isn't part of the > executable sections can be freely modified, and signed ELF files can be > re-signed. >=20 > Having a separate signature for each segment in the program header table > is slightly more complicated than the simplest solution of having one > signature for all program header sections. However, this approach > provides more flexibility going forward. It also accounts for the fact > that we might not want to sign all portions of the file. Finally, as > designed, it allows the file to be modified freely as long as the > runtime behavior isn't affected. >=20 > There is a rather simple design possibility if anyone wanted to go the > signed executable/library route: have an mmap variant with an additional > parameter pointing to the signature would lead to a very simple > modification of the userland dlopen functionality. Normal mmap would > just become a wrapper around the secure variant, which passes in NULL > for the signature (alternatively, you could pass in a default key built > into the local libc, or something similar). Userland shouldn't be trusted to enforce digital signatures. What if someone at link time specifies a non-default RTLD? To enforce digital signatures of userland binaries/libraries, the ELF image activator should be modified to verify the DT_NEEDED entries. >=20 > =3D=3D Conclusion =3D=3D >=20 > This seems like a good point in the design space: it doesn't break > anything, it doesn't require massive changes or rearchitecting of > anything, it provides everything I want to provide now, and it leaves > the door open to things people might want to do in the future. >=20 > Please provide feedback, comments, and suggestions. >=20 >=20 > [0]: There actually is at least one example of something like this of > which I'm aware. The Intel C Compiler (icc, "proton" by Intel internal > naming) has an interprocedural optimization mode which produces .o files > containing the compiler's intermediate representation in a special > section as well as object code in the usual sections (incidentally, in > the distant past, icc would actually produce separate .o and .il files; > this was later changed to the ELF metadata solution, for the very reason > that it complicated build scripts quite a bit). This allows "normal" > compilers and compilation modes to use the object code, while icc uses > the intermediate representation. >=20 The only other major thing to discuss is supporting public key chaining. Ideally, digital signature support should also support chaining multiple keys (similar to X.509 PKI). If the accepted solution supported cert chaining, then the solution would be more modular. I don't want to go down the route of the SSL/TLS PKI mess, but supporting chaining is a must in some enterprise environments. If we were to support chaining, we could even stuff the pubkey half of the key material into another ELF section, so that if a key becomes compromised, the old pubkey can be revoked from the trust store and a new binary can be generated with new key material. The trusted root doesn't need to be cycled as often. HardenedBSD, for example, distributes the pubkey that corresponds to the signing privkey inside the update tarball for binary updates[1]. This allows us to change key material often if desired without the user even noticing. [1]: https://hardenedbsd.org/article/shawn-webb/2015-12-31/introducing-hard= enedbsds-new-binary-updater Caveat with the above-linked article: hbsd-update has undergone additional changes not reflected in the original article. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --2iuyerr5quyn3eol Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAljZW+0ACgkQaoRlj1JF bu4IhxAAlmF0xnGQFGomaYjB2BKc0+oqgawlJ6Xeib0UVSVAl8U3DhnyOlTduTvC 7Olhu3XUqiVd2ggJwihlQJSZ0oO0HTqMx1bMm9uBFDpOQEzLA460ZITOIRiUprzl bEgiOTY4yGWtM9MkQSqhCZFomXqhVNOY4OxLNXp0eXmysug/gaNu9JQD6N2t14Hx e2dCBVAgitK9D/qbYYrxBo53wFQYCTqlQuzEBNG+V8rACrV4mIYFl9T07et02T9/ fMGyZCT/nnsBwyKawKK/M+y7PPnDNm8X5PeWhOgwRm9yfzg3qWRmTMak3NLP5C1+ hqvobjTZcFA2wdRy6Ur8kHcEtSPR7fwgljFzNsK1usLCMBfKZkM19/iTHgdno70l S6gvd4ZKlQb9MgBx2ykBIbC18JfT7mubxQTav09nPB/tk1t+L2HF5MjE8Wx0+eeK YqJSxD3SyH3Q3zixFti5rePQKxlwhHOOZzQR443e7TuPC9iWtIn5rz1ON5xPsHZS jY78nmU1w0DvZoKc/WaMOQWhLaR9P7Oh3T7Urwsa+W0RDmYoZIHvqf3lOb+Kq+Wv SHSNe8EwQvUzIm/aF47G+cNiwHhocxWVWTqpFemWMzyiAxBV0YGFELzvMiYiL1YP logTqsSKQeXYhdL30J7I5Rf2eMXjCNmaFhh9Z6/3qDq9yPu1XLg= =C5y9 -----END PGP SIGNATURE----- --2iuyerr5quyn3eol-- From owner-freebsd-hackers@freebsd.org Mon Mar 27 19:53:37 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03569D20C09; Mon, 27 Mar 2017 19:53:37 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id CBB88EF1; Mon, 27 Mar 2017 19:53:36 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f] (unknown [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 435EA16E9; Mon, 27 Mar 2017 19:53:30 +0000 (UTC) Subject: Re: Proposal for a design for signed kernel/modules/etc To: Shawn Webb References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170327183735.uokjhjaafkawc2id@mutt-hbsd> Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org From: Eric McCorkle Message-ID: <0943546b-2dcd-597b-e000-38926e55bc1d@metricspace.net> Date: Mon, 27 Mar 2017 15:53:26 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327183735.uokjhjaafkawc2id@mutt-hbsd> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="os7uND0ljmwv50huqBfTW7U8F7ROtLvM1" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 19:53:37 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --os7uND0ljmwv50huqBfTW7U8F7ROtLvM1 Content-Type: multipart/mixed; boundary="dVQm9PL85J3IvOTUk061HHwJiacf7UQHs"; protected-headers="v1" From: Eric McCorkle To: Shawn Webb Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Message-ID: <0943546b-2dcd-597b-e000-38926e55bc1d@metricspace.net> Subject: Re: Proposal for a design for signed kernel/modules/etc References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170327183735.uokjhjaafkawc2id@mutt-hbsd> In-Reply-To: <20170327183735.uokjhjaafkawc2id@mutt-hbsd> --dVQm9PL85J3IvOTUk061HHwJiacf7UQHs Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/27/2017 14:37, Shawn Webb wrote: > Hey Eric, >=20 > Thank you for writing this! ELF binary signing has been on my > ever-growing list of things to research and develop. If you'd like help= , > please let me know. I'll probably spin up a branch on my github in the near future. > As git has shown, having a modular/configurable crypto interface is the= > best route. Right now, git is stuck using SHA1 because they didn't > support users being able to choose which hashing algorithm to use. Oh yes! And all current pubkey crypto has an expiration date probably between a decade and a century from now. > You might want to take a look at Microsoft's Authenticode. Microsoft > made some mistakes early on that allowed attackers to easily trojan > signed binaries. Your proposal up to this point makes those same > mistakes. It's been a few years since I researched Authenticode, so I > don't have any links or documentation handy. >=20 > The conclusion Microsoft came to is that the file as a whole must be > signed, including offset metadata. Essentially, you'd determine how > large the .sig section needs to be ahead of time, create it and fill it= > with zeros, then sign the whole file, stuffing the signature in the > zeroed .sig section. Same concept as calculating checksums of ICMP > packets. >=20 > This prevents attackers from modifying critical pieces of metadata, > pointing them to maliciuos payloads. It also prevents attackers from > appending malicious data to the end of a loadable segment (something > Authenticode suffered from early on). Yeah, now that I think about it, there's various forms of evil you could pull off. I don't doubt that someone *could* design a scheme that would defeat all these attacks, but... is it worth it? Probably not. Better to just sign the whole file and seal off all the attack vectors, on second thought. > Userland shouldn't be trusted to enforce digital signatures. What if > someone at link time specifies a non-default RTLD? To enforce digital > signatures of userland binaries/libraries, the ELF image activator > should be modified to verify the DT_NEEDED entries. True, I was assuming dlopen just mmaps everything... (I was getting hungry by that point). As with the whole-file sigs, it would probably be better just to have a single syscall that securely loads dynamic libraries (but that's out-of-scope for now). >=20 > The only other major thing to discuss is supporting public key chaining= =2E > Ideally, digital signature support should also support chaining multipl= e > keys (similar to X.509 PKI). If the accepted solution supported cert > chaining, then the solution would be more modular. I don't want to go > down the route of the SSL/TLS PKI mess, but supporting chaining is a > must in some enterprise environments. >=20 > If we were to support chaining, we could even stuff the pubkey half of > the key material into another ELF section, so that if a key becomes > compromised, the old pubkey can be revoked from the trust store and a > new binary can be generated with new key material. The trusted root > doesn't need to be cycled as often. HardenedBSD, for example, > distributes the pubkey that corresponds to the signing privkey inside > the update tarball for binary updates[1]. This allows us to change key > material often if desired without the user even noticing. I can see a two-tiered scheme where you have a "vendor key", which is used to sign various "application keys", which are used to sign the kernel and modules. The signed executables can then supply their application keys (signed by the vendor key) to the loader/kernel, which first checks the signature on the key, then on the file. This way, you could generate and use a different key for each build, and sign it with the same vendor key. There's at least three use cases in play here: 1) FreeBSD would want to publish ready-made installations, probably signed by a key they control. This ensures that someone who just wants to install binary packages and go doesn't load any kernel/modules other than the ones coming from FreeBSD. 2) I run an enterprise network, and I want to build everything internally on some build-box, and make sure I only run things that got built there, or that came from FreeBSD. 3) I have a highly secure network that is set up like in (2), except I only allow things I build internally to run. 3) I have my laptop, and I want to build everything on the laptop and make sure I don't boot anything I didn't build. For all of these, I'd want to bake in at least the vendor key(s). I could do this pretty easily by converting the key(s) to a header, then using it in buildworld/kerrnel. For (1), FreeBSD would have a vendor key, which it uses to sign the kernel/modules it publishes, and that would be baked in to those applications. For (2), I'd have my own vendor key for my organization, and I'd use that in addition to FreeBSD's key. This would allow anything signed by either of the two organizations. For (3), I'd only use my own vendor key, and not FreeBSD's, so that only things I build internally can boot. For (4), I theoretically only need one level of key. I'd presumably generate a key pair every buildworld/kernel, then throw away the private key when I finished. OTOH, I'd need to remember to reinstall loader (or GRUB/coreboot, and I might not want to go around flashing my BIOS every time I upgrade) every time, so there's a case for a two-level scheme at which point it's a microcosm of (4). --dVQm9PL85J3IvOTUk061HHwJiacf7UQHs-- --os7uND0ljmwv50huqBfTW7U8F7ROtLvM1 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEzzhiNNveVG6nWjcH1w0wQIFco2cFAljZbbYACgkQ1w0wQIFc o2eVyhAAihBjJebgTcoQQO/d9eTgC1QhQzw98Kd5PqDqskVKjBh49GARLZXG0mvg 3ayZE6tmL5qJjJkJirqr9IDn3bfCIbBhv4Mz8fA5dj9uiu+H/ph0FZBlmntejzMe yFDLc+vEKcCB2OJQyf7X9XiEOPWoZDmZKx1Wb2T6OQNZ7n+Ud9Rnbw/wDjgzCgbI zeN21MLI1+94RrAyGlf83uQAd6TDIkI8BV99+mA/6sZKEHL4AdsOfmzwt/XxMYxW oXqednrF+fi6fy8j6t0INERXRkS9lg1i7xUdEOs6Sr30nimPaNyDicgm/W2YpJOa /wWtRBh3Dt0A/GRfa+hODPy5dPmB7OMF0OIqfH0VsmwhdQf0C/fMvsuqs5yIcjvX a8dlBVqDCyCfGI17w6UhXyviGNlcW/vQXc9XiibHFBwHd+uWMBF32W0HB4ecEvJA bALiuSGthBFUKL/7vT3a07eAWlAdEOtETANrionXY/AIw1tpbaJypjuKL4r092qB PKoBbnCSwtl3SPPIxAbzKzX4CgFaBafTMruM0R8QNPolTZ70de/hMEcWxTt0kiP8 iQtGNdlire+KRe2+3dQ3LUh5gIAne1wIBCNtuTyiHEiv0340JX9KGJUgBgGQobkO GSIN4mkUQFbI7926jYcFM62Qk61+2eFv68j1PXIacwXBlEd/bkw= =NkKn -----END PGP SIGNATURE----- --os7uND0ljmwv50huqBfTW7U8F7ROtLvM1-- From owner-freebsd-hackers@freebsd.org Mon Mar 27 20:43:41 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A16AED20F7E for ; Mon, 27 Mar 2017 20:43:41 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-211-172.reflexion.net [208.70.211.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 50A3B8AD for ; Mon, 27 Mar 2017 20:43:40 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 31255 invoked from network); 27 Mar 2017 20:43:39 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 27 Mar 2017 20:43:39 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.30.2) with SMTP; Mon, 27 Mar 2017 16:43:39 -0400 (EDT) Received: (qmail 29684 invoked from network); 27 Mar 2017 20:43:39 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 Mar 2017 20:43:39 -0000 Received: from [192.168.1.111] (c-67-170-167-181.hsd1.or.comcast.net [67.170.167.181]) by iron2.pdx.net (Postfix) with ESMTPSA id CE981EC81E0; Mon, 27 Mar 2017 13:43:38 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD From: Mark Millard In-Reply-To: <20170327164905.GN43712@kib.kiev.ua> Date: Mon, 27 Mar 2017 13:43:38 -0700 Cc: Steven Hartland , "freebsd-hackers@freebsd.org" , "K. Macy" Content-Transfer-Encoding: quoted-printable Message-Id: <065E68C9-C9D5-4702-98A0-42E1A4FA8187@dsl-only.net> References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 20:43:41 -0000 On 2017-Mar-27, at 9:49 AM, Konstantin Belousov = wrote: > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: >> On 27/03/2017 17:18, Konstantin Belousov wrote: . . . >> I noticed in vm_map_copy_entry the following: >> /* >> * We don't want to make writeable wired pages=20 >> copy-on-write. >> * Immediately copy these pages into the new map by=20= >> simulating >> * page faults. The new pages are pageable. >> */ >> vm_fault_copy_entry(dst_map, src_map, dst_entry, = src_entry, >> fork_charge); >>=20 >> I wondered if I could use vm_fault_copy_entry to force the copy on = fork? > No, the vm_fault_copy_entry() only works with wired entries, e.g. it = cannot > page in not yet touched page, and the result is also wired. . . . I'm confused by the comment vs. the above note: Comment: The new pages are pageable. This note: the result is also wired So pagable wired pages? Incorrect comment? Comment that needs more context specified so the interpretation is clearer? =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-hackers@freebsd.org Mon Mar 27 23:16:40 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2AC0D200F6 for ; Mon, 27 Mar 2017 23:16:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x235.google.com (mail-wr0-x235.google.com [IPv6:2a00:1450:400c:c0c::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6013A1BD for ; Mon, 27 Mar 2017 23:16:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x235.google.com with SMTP id w43so65989206wrb.0 for ; Mon, 27 Mar 2017 16:16:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=PZFcqGUpLQmIzA2FxR7xd8o80B3Pnkxu31kGH2lnmZ0=; b=QawtVmiM2deKrk8YeSSz7uwTm2p2IFH5j6PUYEqC7XY12ZPUWae+aFe6jC8CqFQujO i0RDM+GJueAI9JsCGJw+MibSSr2T4GuVv7ifm12B7Fu4UE6ibpIueQnANBkBuiNn3uT2 SQ9rK5Pf1s/TVxlzkbM7tlQUMORXaACwbCuuXAQu/lpRpSOYaa9wMZTp6dEitHS+HSk8 2pGg7tTZGexRwznOWrHAkAm04f1Ch1dfjQxD6WGYfvEDNIN70YuLjGQMY/mScPPtJgFv eJMC/WydXlfoS3mw6j+HGADGmaooTCzHBo3zpaxggUCM+O8nZigLay77lDbdBEex3Zng TsSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=PZFcqGUpLQmIzA2FxR7xd8o80B3Pnkxu31kGH2lnmZ0=; b=RkkUhbRRGupOHu4c7+S9L/Ba2uiroPBAll3EygppqmwM6gm1yFwFZ+XxJgfnpiBokl ZCGnsjMp2fsDzrxdYggDV37dI5actWcmVPFLO8pGhjgq8MRpjPMo3MGLJgWp53s+EOVM tSLcKlcstuvRUKwjWbuPjGZCRemz1jZqvMZFQBylXbWHyXJLaFaIm2oro+8qven6gyLb 7ufChODf5cbR/gxIJuBe/0L1GaGvthHwrAyKPHzLivoW9s8HAYCeIVE0uuw4jLpQq2Fp snrqQAwDjcsi23wq8Jeq8BDMuPWNx7VKR5Gv3VfNyTyfTo0czZeEury4XyAQI1za6TkN +bUw== X-Gm-Message-State: AFeK/H0O9wmxQYKu9sMH0RefApoA+JHzOgMdoiUGBn+kcpyJZMsx7CDerOomeidOqUzPLGOk X-Received: by 10.28.153.20 with SMTP id b20mr6588118wme.76.1490656598475; Mon, 27 Mar 2017 16:16:38 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id u145sm1237481wmu.1.2017.03.27.16.16.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 16:16:37 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: Date: Tue, 28 Mar 2017 00:16:38 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327164905.GN43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 23:16:40 -0000 On 27/03/2017 17:49, Konstantin Belousov wrote: > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: >> On 27/03/2017 17:18, Konstantin Belousov wrote: >>> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >>>> OK now the similar but unrelated issue with signal stacks is solved I've >>>> moved back to the initial issue. >>>> >>>> I've made some progress with a reproduction case as detailed here: >>>> https://github.com/golang/go/issues/15658#issuecomment-288747812 >>>> >>>> In short it seems that having a running child, while the parent runs GC, >>>> is some how responsible for memory corruption in the parent. >>>> >>>> The reason I believe this is if I run the same GC in the parent after >>>> the child exits instead of while its running, I've been unable to >>>> reproduce the issue. >>>> >>>> As the memory segments are COW then the issue might be in VM subsystem. >>> Well, it might be, but it is a strange corruption mode to believe. >> Indeed, but would you agree the evidence seems to indicate that this may >> be the case, as otherwise I would have expected that running the GC >> after the child process has exited would have zero impact on the issue. >>>> In order to confirm / deny this I was wondering if there was a way to >>>> force a full copy of all segments for the child instead of using the COW >>>> optimisation. >>> No, there is no. By design, copying only occurs on faults, when VM >>> detects that the map entry needs copying. Doing the actual copy at fork >>> time would require writing a lot of new code. >> I noticed in vm_map_copy_entry the following: >> /* >> * We don't want to make writeable wired pages >> copy-on-write. >> * Immediately copy these pages into the new map by >> simulating >> * page faults. The new pages are pageable. >> */ >> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, >> fork_charge); >> >> I wondered if I could use vm_fault_copy_entry to force the copy on fork? > No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot > page in not yet touched page, and the result is also wired. > >>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? >> Yes it does, I don't currently have i386 machine to test with, I'm >> assuming testing i386 on amd64 kernel, would likely not have any effect. > Only if the bug is in kernel and not in the go runtime. I am still not > convinced that the kernel is the culprit. > >>> Another blind experiment to try is to comment out call to >>> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if >>> it changes anything. >> I'll do that shortly. Still crashed with vm_object_collapse commented out, here's the parent procstat -v: PID START END PRT RES PRES REF SHD FLAG TP PATH 69713 0x400000 0x70e000 r-x 306 601 3 1 CN-- vn /root/golang/src/test5/test5 69713 0x70e000 0x951000 r-- 263 601 3 1 CN-- vn /root/golang/src/test5/test5 69713 0x951000 0x988000 rw- 31 0 1 0 C--- vn /root/golang/src/test5/test5 69713 0x988000 0x9ab000 rw- 18 18 1 0 C--- df 69713 0x800951000 0x800b51000 rw- 41 41 1 0 C--- df 69713 0x800b51000 0x800c21000 rw- 27 27 1 0 C--- df 69713 0x800c21000 0x800c31000 rw- 16 16 1 0 C--- df 69713 0x800c31000 0x800c71000 rw- 1 1 1 0 C--- df 69713 0x800c71000 0x800cf1000 rw- 5 5 1 0 C--- df 69713 0x800cf1000 0x800d31000 rw- 1 1 1 0 CN-- df 69713 0x800d31000 0x800d71000 rw- 1 1 1 0 C--- df 69713 0x800d71000 0x800e31000 rw- 3 3 1 0 C--- df 69713 0x800e31000 0x800eb1000 rw- 3 3 1 0 C--- df 69713 0x800eb1000 0x800ef1000 rw- 2 2 1 0 C--- df 69713 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df 69713 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df 69713 0xc41fff8000 0xc420200000 rw- 267 267 1 0 C--- df 69713 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 69713 0x7ffffffff000 0x800000000000 r-x 1 1 27 0 ---- ph Regards Steve From owner-freebsd-hackers@freebsd.org Mon Mar 27 23:32:39 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BFA3D20688 for ; Mon, 27 Mar 2017 23:32:39 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22e.google.com (mail-wr0-x22e.google.com [IPv6:2a00:1450:400c:c0c::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B948DDB6 for ; Mon, 27 Mar 2017 23:32:38 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22e.google.com with SMTP id u1so79470916wra.2 for ; Mon, 27 Mar 2017 16:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=1+gsSINhaQYlKt57S86iI7D2L0GICoBqhcOWc7J699s=; b=hrFRiW0kyMpDyE4EgIxfrRbTGD67em3D05oCsGJzDrtSEckd8DN7C0wWHY6RZ1Kbx8 1CDgpN7QBP26+STXdvex/pAWekGPV8I/CbbDl2ejrVQ/g+ZdlfrVYD8vCMIYyXO//tnA 5FrfG899sIP30ChjQXbScli8Nw/l3fbz4SUkuEuVZrHBNdkIWYPEbKqOBdfLj5hrgHfS 98UXiB/e48itdEGGy6nXp8rKlJFTMgKbS17KcW7V/ldr4QHEed789sASooD4f5BYt6Mq i89NeyqThySd9U18PudpwkR3MqJd7PYH7StNhw0SszlH89i2iJqvHjPkA6AW/8Cximf+ veXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=1+gsSINhaQYlKt57S86iI7D2L0GICoBqhcOWc7J699s=; b=ilUfDxFtOo76ptKkL2TrAvXQ8F9DxuCkIfnqvGK0+IejFVVKgk+Ai+bIKjFvyB0CpN t2/z7FCoUyEiXsEhjYjPreVFBTlHsZeOco+f0CckDjvb1WGIQ9ID6j1202WOy4beacyI QF7ntLSU6pl+mkmdrWW5sNQWJScc17taHVKedL5yh7a5Dc2oq4KQVyS/fLdss5xapSOf cHsDKH6ezd7XHPSBcSmP55wMPBG+924bxI88+K0cICUHp9hSXwYXzW5MinYsvQ/lA2QW 4zCUIUMhXDiNkqpxVy1ffnnm7cDvC9NXRn3cZjXrrmVav2H68TEef/4/Ft5KK5F7JEu/ 3A9A== X-Gm-Message-State: AFeK/H1qLj6G5YjifCsa7tBRd8FcpvIlVdhrDvzxlVjsFI11Lm7ArLs3t/aeH1m2gElIYYQW X-Received: by 10.223.156.2 with SMTP id f2mr22096235wrc.176.1490657556797; Mon, 27 Mar 2017 16:32:36 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id l90sm1234233wmi.25.2017.03.27.16.32.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 16:32:35 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> Date: Tue, 28 Mar 2017 00:32:37 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327164905.GN43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 23:32:39 -0000 On 27/03/2017 17:49, Konstantin Belousov wrote: > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: >> On 27/03/2017 17:18, Konstantin Belousov wrote: >>> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >>>> OK now the similar but unrelated issue with signal stacks is solved I've >>>> moved back to the initial issue. >>>> >>>> I've made some progress with a reproduction case as detailed here: >>>> https://github.com/golang/go/issues/15658#issuecomment-288747812 >>>> >>>> In short it seems that having a running child, while the parent runs GC, >>>> is some how responsible for memory corruption in the parent. >>>> >>>> The reason I believe this is if I run the same GC in the parent after >>>> the child exits instead of while its running, I've been unable to >>>> reproduce the issue. >>>> >>>> As the memory segments are COW then the issue might be in VM subsystem. >>> Well, it might be, but it is a strange corruption mode to believe. >> Indeed, but would you agree the evidence seems to indicate that this may >> be the case, as otherwise I would have expected that running the GC >> after the child process has exited would have zero impact on the issue. >>>> In order to confirm / deny this I was wondering if there was a way to >>>> force a full copy of all segments for the child instead of using the COW >>>> optimisation. >>> No, there is no. By design, copying only occurs on faults, when VM >>> detects that the map entry needs copying. Doing the actual copy at fork >>> time would require writing a lot of new code. >> I noticed in vm_map_copy_entry the following: >> /* >> * We don't want to make writeable wired pages >> copy-on-write. >> * Immediately copy these pages into the new map by >> simulating >> * page faults. The new pages are pageable. >> */ >> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, >> fork_charge); >> >> I wondered if I could use vm_fault_copy_entry to force the copy on fork? > No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot > page in not yet touched page, and the result is also wired. > >>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? >> Yes it does, I don't currently have i386 machine to test with, I'm >> assuming testing i386 on amd64 kernel, would likely not have any effect. > Only if the bug is in kernel and not in the go runtime. I am still not > convinced that the kernel is the culprit. > >>> Another blind experiment to try is to comment out call to >>> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if >>> it changes anything. >> I'll do that shortly. >>> What could be quite interesting is to look at the parent and possibly >>> child address map after the error occured, using procstat -v. At >>> least for parent, this should be relatively easy to set up, just make >>> go runtime spin or pause on panic, instead of exiting, and then use >>> procstat. >> Here's both parent and child after a failure in the parent, which I obtained by putting the child in a nanosleep loop and only after successful GC call I send SIGTERM the child and reap it. procstat -v 53832 61121 PID START END PRT RES PRES REF SHD FLAG TP PATH 53832 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn /root/golang/src/test5/test5 53832 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn /root/golang/src/test5/test5 53832 0x951000 0x988000 rw- 31 0 1 0 C--- vn /root/golang/src/test5/test5 53832 0x988000 0x9ab000 rw- 18 0 1 0 C--- df 53832 0x800951000 0x800b51000 rw- 41 0 1 0 C--- df 53832 0x800b51000 0x800c21000 rw- 26 0 1 0 C--- df 53832 0x800c21000 0x800c71000 rw- 18 0 1 0 C--- df 53832 0x800c71000 0x800cb1000 rw- 1 0 1 0 C--- df 53832 0x800cb1000 0x800cf1000 rw- 2 0 1 0 C--- df 53832 0x800cf1000 0x800d71000 rw- 3 0 1 0 C--- df 53832 0x800d71000 0x800db1000 rw- 1 0 1 0 C--- df 53832 0x800db1000 0x800e71000 rw- 3 0 1 0 C--- df 53832 0x800e71000 0x800eb1000 rw- 1 1 1 0 ---- df 53832 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 53832 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 53832 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df 53832 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df 53832 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph 61121 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn /root/golang/src/test5/test5 61121 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn /root/golang/src/test5/test5 61121 0x951000 0x988000 rw- 31 0 2 1 CN-- vn /root/golang/src/test5/test5 61121 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df 61121 0x800951000 0x800b51000 rw- 41 41 2 1 CN-- df 61121 0x800b51000 0x800c21000 rw- 26 26 2 1 CN-- df 61121 0x800c21000 0x800c71000 rw- 18 18 2 1 CN-- df 61121 0x800c71000 0x800cb1000 rw- 1 1 2 1 CN-- df 61121 0x800cb1000 0x800cf1000 rw- 2 2 2 1 CN-- df 61121 0x800cf1000 0x800d71000 rw- 3 3 2 1 CN-- df 61121 0x800d71000 0x800db1000 rw- 1 1 2 1 CN-- df 61121 0x800db1000 0x800e71000 rw- 3 3 2 1 CN-- df 61121 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 61121 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 61121 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df 61121 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df 61121 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph Should the parent have lost the COW flag to the region starting at 0x800e71000? Regards Steve Regards Steve From owner-freebsd-hackers@freebsd.org Tue Mar 28 07:54:53 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55FBCD207DF for ; Tue, 28 Mar 2017 07:54:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1BDBB51; Tue, 28 Mar 2017 07:54:52 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2S7skg9018417 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Mar 2017 10:54:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2S7skg9018417 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2S7skDf018416; Tue, 28 Mar 2017 10:54:46 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Mar 2017 10:54:46 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170328075446.GO43712@kib.kiev.ua> References: <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 07:54:53 -0000 On Mon, Mar 27, 2017 at 07:17:20PM +0100, Steven Hartland wrote: > Here's the output after its logged the errors. > > procstat -v 36159 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 36159 0x400000 0x70d000 r-x 308 588 3 1 CN-- vn > /root/golang/src/test5/test5 > 36159 0x70d000 0x94e000 r-- 248 588 3 1 CN-- vn > /root/golang/src/test5/test5 > 36159 0x94e000 0x985000 rw- 31 0 1 0 C--- vn > /root/golang/src/test5/test5 > 36159 0x985000 0x9a8000 rw- 18 18 1 0 C--- df > 36159 0x80094e000 0x8009ee000 rw- 35 35 1 0 C--- df > 36159 0x8009ee000 0x800abe000 rw- 28 28 1 0 C--- df > 36159 0x800abe000 0x800ace000 rw- 16 16 1 0 C--- df > 36159 0x800ace000 0x800b0e000 rw- 2 2 1 0 CN-- df > 36159 0x800b0e000 0x800b4e000 rw- 2 2 1 0 C--- df > 36159 0x800b4e000 0x800b8e000 rw- 2 2 1 0 C--- df > 36159 0x800b8e000 0x800c0e000 rw- 3 3 1 0 C--- df > 36159 0x800c0e000 0x800c4e000 rw- 2 2 1 0 C--- df > 36159 0x800c4e000 0x800c8e000 rw- 1 1 1 0 CN-- df > 36159 0x800c8e000 0x800cce000 rw- 1 1 1 0 C--- df > 36159 0x800cce000 0x800d0e000 rw- 1 1 1 0 CN-- df > 36159 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df > 36159 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df > 36159 0xc41fff8000 0xc420200000 rw- 258 258 1 0 C--- df > 36159 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df > 36159 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph > > I've left it running in case you want any more info. At which address the word is located where an invalid value was detected ? From owner-freebsd-hackers@freebsd.org Tue Mar 28 07:56:55 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9684DD209E6 for ; Tue, 28 Mar 2017 07:56:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0EFC6D27; Tue, 28 Mar 2017 07:56:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2S7um2h019487 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Mar 2017 10:56:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2S7um2h019487 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2S7ul0q019486; Tue, 28 Mar 2017 10:56:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Mar 2017 10:56:47 +0300 From: Konstantin Belousov To: Mark Millard Cc: Steven Hartland , "freebsd-hackers@freebsd.org" , "K. Macy" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170328075647.GP43712@kib.kiev.ua> References: <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <065E68C9-C9D5-4702-98A0-42E1A4FA8187@dsl-only.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <065E68C9-C9D5-4702-98A0-42E1A4FA8187@dsl-only.net> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 07:56:55 -0000 On Mon, Mar 27, 2017 at 01:43:38PM -0700, Mark Millard wrote: > On 2017-Mar-27, at 9:49 AM, Konstantin Belousov wrote: > > > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: > >> On 27/03/2017 17:18, Konstantin Belousov wrote: > . . . > >> I noticed in vm_map_copy_entry the following: > >> /* > >> * We don't want to make writeable wired pages > >> copy-on-write. > >> * Immediately copy these pages into the new map by > >> simulating > >> * page faults. The new pages are pageable. > >> */ > >> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, > >> fork_charge); > >> > >> I wondered if I could use vm_fault_copy_entry to force the copy on fork? > > No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot > > page in not yet touched page, and the result is also wired. > . . . > > I'm confused by the comment vs. the above note: > > Comment: The new pages are pageable. > This note: the result is also wired Yes, my note about destination state is not true. Still, the code cannot page in source pages, and the intent of the note above was that shadow chain for the destination entry will be invalid even if required page-ins were done. > > So pagable wired pages? Incorrect comment? Comment > that needs more context specified so the interpretation > is clearer? > > > === > Mark Millard > markmi at dsl-only.net > From owner-freebsd-hackers@freebsd.org Tue Mar 28 07:59:05 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4679CD20D1A for ; Tue, 28 Mar 2017 07:59:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E2406FF3; Tue, 28 Mar 2017 07:59:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2S7wxwc019520 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Mar 2017 10:58:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2S7wxwc019520 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2S7wxSS019519; Tue, 28 Mar 2017 10:58:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Mar 2017 10:58:59 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170328075859.GQ43712@kib.kiev.ua> References: <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 07:59:05 -0000 On Tue, Mar 28, 2017 at 12:32:37AM +0100, Steven Hartland wrote: > On 27/03/2017 17:49, Konstantin Belousov wrote: > > On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: > >> On 27/03/2017 17:18, Konstantin Belousov wrote: > >>> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: > >>>> OK now the similar but unrelated issue with signal stacks is solved I've > >>>> moved back to the initial issue. > >>>> > >>>> I've made some progress with a reproduction case as detailed here: > >>>> https://github.com/golang/go/issues/15658#issuecomment-288747812 > >>>> > >>>> In short it seems that having a running child, while the parent runs GC, > >>>> is some how responsible for memory corruption in the parent. > >>>> > >>>> The reason I believe this is if I run the same GC in the parent after > >>>> the child exits instead of while its running, I've been unable to > >>>> reproduce the issue. > >>>> > >>>> As the memory segments are COW then the issue might be in VM subsystem. > >>> Well, it might be, but it is a strange corruption mode to believe. > >> Indeed, but would you agree the evidence seems to indicate that this may > >> be the case, as otherwise I would have expected that running the GC > >> after the child process has exited would have zero impact on the issue. > >>>> In order to confirm / deny this I was wondering if there was a way to > >>>> force a full copy of all segments for the child instead of using the COW > >>>> optimisation. > >>> No, there is no. By design, copying only occurs on faults, when VM > >>> detects that the map entry needs copying. Doing the actual copy at fork > >>> time would require writing a lot of new code. > >> I noticed in vm_map_copy_entry the following: > >> /* > >> * We don't want to make writeable wired pages > >> copy-on-write. > >> * Immediately copy these pages into the new map by > >> simulating > >> * page faults. The new pages are pageable. > >> */ > >> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, > >> fork_charge); > >> > >> I wondered if I could use vm_fault_copy_entry to force the copy on fork? > > No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot > > page in not yet touched page, and the result is also wired. > > > >>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? > >> Yes it does, I don't currently have i386 machine to test with, I'm > >> assuming testing i386 on amd64 kernel, would likely not have any effect. > > Only if the bug is in kernel and not in the go runtime. I am still not > > convinced that the kernel is the culprit. > > > >>> Another blind experiment to try is to comment out call to > >>> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if > >>> it changes anything. > >> I'll do that shortly. > >>> What could be quite interesting is to look at the parent and possibly > >>> child address map after the error occured, using procstat -v. At > >>> least for parent, this should be relatively easy to set up, just make > >>> go runtime spin or pause on panic, instead of exiting, and then use > >>> procstat. > >> > Here's both parent and child after a failure in the parent, which I > obtained by putting the child in a nanosleep loop and only after > successful GC call I send SIGTERM the child and reap it. > > procstat -v 53832 61121 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 53832 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn > /root/golang/src/test5/test5 > 53832 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn > /root/golang/src/test5/test5 > 53832 0x951000 0x988000 rw- 31 0 1 0 C--- vn > /root/golang/src/test5/test5 > 53832 0x988000 0x9ab000 rw- 18 0 1 0 C--- df > 53832 0x800951000 0x800b51000 rw- 41 0 1 0 C--- df > 53832 0x800b51000 0x800c21000 rw- 26 0 1 0 C--- df > 53832 0x800c21000 0x800c71000 rw- 18 0 1 0 C--- df > 53832 0x800c71000 0x800cb1000 rw- 1 0 1 0 C--- df > 53832 0x800cb1000 0x800cf1000 rw- 2 0 1 0 C--- df > 53832 0x800cf1000 0x800d71000 rw- 3 0 1 0 C--- df > 53832 0x800d71000 0x800db1000 rw- 1 0 1 0 C--- df > 53832 0x800db1000 0x800e71000 rw- 3 0 1 0 C--- df > 53832 0x800e71000 0x800eb1000 rw- 1 1 1 0 ---- df > 53832 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 53832 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 53832 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df > 53832 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df > 53832 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph > 61121 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn > /root/golang/src/test5/test5 > 61121 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn > /root/golang/src/test5/test5 > 61121 0x951000 0x988000 rw- 31 0 2 1 CN-- vn > /root/golang/src/test5/test5 > 61121 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df > 61121 0x800951000 0x800b51000 rw- 41 41 2 1 CN-- df > 61121 0x800b51000 0x800c21000 rw- 26 26 2 1 CN-- df > 61121 0x800c21000 0x800c71000 rw- 18 18 2 1 CN-- df > 61121 0x800c71000 0x800cb1000 rw- 1 1 2 1 CN-- df > 61121 0x800cb1000 0x800cf1000 rw- 2 2 2 1 CN-- df > 61121 0x800cf1000 0x800d71000 rw- 3 3 2 1 CN-- df > 61121 0x800d71000 0x800db1000 rw- 1 1 2 1 CN-- df > 61121 0x800db1000 0x800e71000 rw- 3 3 2 1 CN-- df > 61121 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 61121 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 61121 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df > 61121 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df > 61121 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph > > Should the parent have lost the COW flag to the region starting at > 0x800e71000? Note that the region is absent in the child, so most likely this is a new mmaping occured in the parent after fork. Again, what is the address where an invalid value was detected ? From owner-freebsd-hackers@freebsd.org Tue Mar 28 08:23:27 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C08AD1BFC4 for ; Tue, 28 Mar 2017 08:23:27 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCF1B95F for ; Tue, 28 Mar 2017 08:23:26 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x232.google.com with SMTP id w43so79022947wrb.0 for ; Tue, 28 Mar 2017 01:23:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=1x9iC+woiHK49a0TwycHH5fU2kiKmpYCKwBII7rsNDU=; b=VJR5ZfhhXRB0R/P2Co+k1ozosj0en4XBWHr+TcsOq4p69xYMmn3UujSOHnUBXvUFh4 SVN1qENXH77dgJ36ZjI/Voc4WXfMa9DtLVXPjthR/GAhk/wd1YgQ8snSzneSyCqu6fWe SOdFlRiXSvQ5Lg4h7bRk7OAfQw7n9irmjX9uXiL2nVfdI9/euphJJrL4B27EnNxLTkK1 VKVX5hTIzWWlZOa4VrRyf5EUC7cmz/XwmhIW6bvafnIS3NG7Ph9N05OnfOd8KaEiBW40 rtmoINn+08CY3ClOpQvSiMcBDK+wqM2FB5BmPQWbItJn39+m5eYrXEyXl8T5yPqRZ5pH ZSlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=1x9iC+woiHK49a0TwycHH5fU2kiKmpYCKwBII7rsNDU=; b=G1RwtFxTmbwr37P6TgpFb0aVPUClqhb9mDIvcLGqztdckcgyMkIuVswoCnOyIm3rgQ UhCL4LcQxh6Fc0u61IQIsENXgRCDbugfzEL8I7kYf41T9VjI/3N+B4n3h/VJG83CKRDm uc1mYKBQ7ns9GT9AF80BrSEkexwjgzdcNnHyPhJH4CEvlHDNKPKe88Dpxf7QYsGtJe1M A4SIS3WFYTTUwaMClZ3OTlH00OOS1ENOQQ4vN52kwEQNpBjJSDsU4jwBJgbFvWcJsBSO bOJqoEFHG7QFljx+8mztkKDRE2Bnl9S7jENpmlhrLKyeMdDAa9Vrzn24BF/Bn5kF+roS 9gPg== X-Gm-Message-State: AFeK/H15a4KRig3KcQHiNWfKN8HqCdN1U6U3EHRn6njl9PdEfegr7PLaTvpJx+b89UW66jb7 X-Received: by 10.28.203.204 with SMTP id b195mr13480600wmg.51.1490689403957; Tue, 28 Mar 2017 01:23:23 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id w2sm2574974wmd.19.2017.03.28.01.23.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 01:23:23 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> Date: Tue, 28 Mar 2017 09:23:24 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170328075859.GQ43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 08:23:27 -0000 On 28/03/2017 08:58, Konstantin Belousov wrote: > On Tue, Mar 28, 2017 at 12:32:37AM +0100, Steven Hartland wrote: >> On 27/03/2017 17:49, Konstantin Belousov wrote: >>> On Mon, Mar 27, 2017 at 05:33:49PM +0100, Steven Hartland wrote: >>>> On 27/03/2017 17:18, Konstantin Belousov wrote: >>>>> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >>>>>> OK now the similar but unrelated issue with signal stacks is solved I've >>>>>> moved back to the initial issue. >>>>>> >>>>>> I've made some progress with a reproduction case as detailed here: >>>>>> https://github.com/golang/go/issues/15658#issuecomment-288747812 >>>>>> >>>>>> In short it seems that having a running child, while the parent runs GC, >>>>>> is some how responsible for memory corruption in the parent. >>>>>> >>>>>> The reason I believe this is if I run the same GC in the parent after >>>>>> the child exits instead of while its running, I've been unable to >>>>>> reproduce the issue. >>>>>> >>>>>> As the memory segments are COW then the issue might be in VM subsystem. >>>>> Well, it might be, but it is a strange corruption mode to believe. >>>> Indeed, but would you agree the evidence seems to indicate that this may >>>> be the case, as otherwise I would have expected that running the GC >>>> after the child process has exited would have zero impact on the issue. >>>>>> In order to confirm / deny this I was wondering if there was a way to >>>>>> force a full copy of all segments for the child instead of using the COW >>>>>> optimisation. >>>>> No, there is no. By design, copying only occurs on faults, when VM >>>>> detects that the map entry needs copying. Doing the actual copy at fork >>>>> time would require writing a lot of new code. >>>> I noticed in vm_map_copy_entry the following: >>>> /* >>>> * We don't want to make writeable wired pages >>>> copy-on-write. >>>> * Immediately copy these pages into the new map by >>>> simulating >>>> * page faults. The new pages are pageable. >>>> */ >>>> vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, >>>> fork_charge); >>>> >>>> I wondered if I could use vm_fault_copy_entry to force the copy on fork? >>> No, the vm_fault_copy_entry() only works with wired entries, e.g. it cannot >>> page in not yet touched page, and the result is also wired. >>> >>>>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? >>>> Yes it does, I don't currently have i386 machine to test with, I'm >>>> assuming testing i386 on amd64 kernel, would likely not have any effect. >>> Only if the bug is in kernel and not in the go runtime. I am still not >>> convinced that the kernel is the culprit. >>> >>>>> Another blind experiment to try is to comment out call to >>>>> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if >>>>> it changes anything. >>>> I'll do that shortly. >>>>> What could be quite interesting is to look at the parent and possibly >>>>> child address map after the error occured, using procstat -v. At >>>>> least for parent, this should be relatively easy to set up, just make >>>>> go runtime spin or pause on panic, instead of exiting, and then use >>>>> procstat. >> Here's both parent and child after a failure in the parent, which I >> obtained by putting the child in a nanosleep loop and only after >> successful GC call I send SIGTERM the child and reap it. >> >> procstat -v 53832 61121 >> PID START END PRT RES PRES REF SHD FLAG >> TP PATH >> 53832 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 53832 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 53832 0x951000 0x988000 rw- 31 0 1 0 C--- vn >> /root/golang/src/test5/test5 >> 53832 0x988000 0x9ab000 rw- 18 0 1 0 C--- df >> 53832 0x800951000 0x800b51000 rw- 41 0 1 0 C--- df >> 53832 0x800b51000 0x800c21000 rw- 26 0 1 0 C--- df >> 53832 0x800c21000 0x800c71000 rw- 18 0 1 0 C--- df >> 53832 0x800c71000 0x800cb1000 rw- 1 0 1 0 C--- df >> 53832 0x800cb1000 0x800cf1000 rw- 2 0 1 0 C--- df >> 53832 0x800cf1000 0x800d71000 rw- 3 0 1 0 C--- df >> 53832 0x800d71000 0x800db1000 rw- 1 0 1 0 C--- df >> 53832 0x800db1000 0x800e71000 rw- 3 0 1 0 C--- df >> 53832 0x800e71000 0x800eb1000 rw- 1 1 1 0 ---- df >> 53832 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 53832 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 53832 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 53832 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df >> 53832 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> 61121 0x400000 0x70e000 r-x 308 601 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 61121 0x70e000 0x951000 r-- 261 601 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 61121 0x951000 0x988000 rw- 31 0 2 1 CN-- vn >> /root/golang/src/test5/test5 >> 61121 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df >> 61121 0x800951000 0x800b51000 rw- 41 41 2 1 CN-- df >> 61121 0x800b51000 0x800c21000 rw- 26 26 2 1 CN-- df >> 61121 0x800c21000 0x800c71000 rw- 18 18 2 1 CN-- df >> 61121 0x800c71000 0x800cb1000 rw- 1 1 2 1 CN-- df >> 61121 0x800cb1000 0x800cf1000 rw- 2 2 2 1 CN-- df >> 61121 0x800cf1000 0x800d71000 rw- 3 3 2 1 CN-- df >> 61121 0x800d71000 0x800db1000 rw- 1 1 2 1 CN-- df >> 61121 0x800db1000 0x800e71000 rw- 3 3 2 1 CN-- df >> 61121 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 61121 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 61121 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 61121 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df >> 61121 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> >> Should the parent have lost the COW flag to the region starting at >> 0x800e71000? > Note that the region is absent in the child, so most likely this is a new > mmaping occured in the parent after fork. Again, what is the address > where an invalid value was detected ? As I stopped the panic before that I couldn't tell so I've re-run with some debug added just before the panic to capture the addresses of the workbuf structure that the issue was detected in, here goes (parent: 62620, child: 98756): workbuf: 0x800b51800 fatal error: workbuf is not empty workbuf: 0x800a72000 fatal error: workbuf is empty workbuf: 0x800a72000 fatal error: workbuf is not empty procstat -v 62620 98756 PID START END PRT RES PRES REF SHD FLAG TP PATH 62620 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn /root/golang/src/test5/test5 62620 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn /root/golang/src/test5/test5 62620 0x951000 0x988000 rw- 31 0 1 0 C--- vn /root/golang/src/test5/test5 62620 0x988000 0x9ab000 rw- 18 0 1 0 C--- df 62620 0x800951000 0x8009f1000 rw- 36 0 1 0 C--- df 62620 0x8009f1000 0x800ac1000 rw- 28 0 1 0 C--- df 62620 0x800ac1000 0x800ad1000 rw- 16 0 1 0 C--- df 62620 0x800ad1000 0x800b91000 rw- 5 0 1 0 C--- df 62620 0x800b91000 0x800bd1000 rw- 2 0 1 0 C--- df 62620 0x800bd1000 0x800c11000 rw- 2 0 1 0 C--- df 62620 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df 62620 0x800c51000 0x800c91000 rw- 1 0 1 0 C--- df 62620 0x800c91000 0x800cd1000 rw- 1 0 1 0 C--- df 62620 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 62620 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 62620 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df 62620 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df 62620 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph 98756 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn /root/golang/src/test5/test5 98756 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn /root/golang/src/test5/test5 98756 0x951000 0x988000 rw- 31 0 2 1 CN-- vn /root/golang/src/test5/test5 98756 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df 98756 0x800951000 0x8009f1000 rw- 36 36 2 1 CN-- df 98756 0x8009f1000 0x800ac1000 rw- 28 28 2 1 CN-- df 98756 0x800ac1000 0x800ad1000 rw- 16 16 2 1 CN-- df 98756 0x800ad1000 0x800b91000 rw- 5 5 2 1 CN-- df 98756 0x800b91000 0x800bd1000 rw- 2 2 2 1 CN-- df 98756 0x800bd1000 0x800c11000 rw- 2 2 2 1 CN-- df 98756 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df 98756 0x800c51000 0x800c91000 rw- 1 1 2 1 CN-- df 98756 0x800c91000 0x800cd1000 rw- 1 1 2 1 CN-- df 98756 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 98756 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 98756 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df 98756 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df 98756 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph Regards Steve From owner-freebsd-hackers@freebsd.org Tue Mar 28 08:38:16 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1AA2D21542 for ; Tue, 28 Mar 2017 08:38:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3E8F23CC; Tue, 28 Mar 2017 08:38:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2S8cABe028383 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Mar 2017 11:38:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2S8cABe028383 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2S8cAdE028382; Tue, 28 Mar 2017 11:38:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Mar 2017 11:38:10 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170328083810.GR43712@kib.kiev.ua> References: <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 08:38:16 -0000 On Tue, Mar 28, 2017 at 09:23:24AM +0100, Steven Hartland wrote: > As I stopped the panic before that I couldn't tell so I've re-run with > some debug added just before the panic to capture the addresses of the > workbuf structure that the issue was detected in, here goes (parent: > 62620, child: 98756): > > workbuf: 0x800b51800 > fatal error: workbuf is not empty > workbuf: 0x800a72000 > fatal error: workbuf is empty > workbuf: 0x800a72000 > fatal error: workbuf is not empty I do not understand. Why do you show several addresses ? Wouldn't the runtime panic after detecting the discrepancy, so there could be only one address ? > > procstat -v 62620 98756 > PID START END PRT RES PRES REF SHD FLAG > TP PATH > 62620 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn > /root/golang/src/test5/test5 > 62620 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn > /root/golang/src/test5/test5 > 62620 0x951000 0x988000 rw- 31 0 1 0 C--- vn > /root/golang/src/test5/test5 > 62620 0x988000 0x9ab000 rw- 18 0 1 0 C--- df > 62620 0x800951000 0x8009f1000 rw- 36 0 1 0 C--- df > 62620 0x8009f1000 0x800ac1000 rw- 28 0 1 0 C--- df > 62620 0x800ac1000 0x800ad1000 rw- 16 0 1 0 C--- df > 62620 0x800ad1000 0x800b91000 rw- 5 0 1 0 C--- df > 62620 0x800b91000 0x800bd1000 rw- 2 0 1 0 C--- df > 62620 0x800bd1000 0x800c11000 rw- 2 0 1 0 C--- df > 62620 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df > 62620 0x800c51000 0x800c91000 rw- 1 0 1 0 C--- df > 62620 0x800c91000 0x800cd1000 rw- 1 0 1 0 C--- df > 62620 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 62620 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 62620 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df > 62620 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df > 62620 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph > 98756 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn > /root/golang/src/test5/test5 > 98756 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn > /root/golang/src/test5/test5 > 98756 0x951000 0x988000 rw- 31 0 2 1 CN-- vn > /root/golang/src/test5/test5 > 98756 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df > 98756 0x800951000 0x8009f1000 rw- 36 36 2 1 CN-- df > 98756 0x8009f1000 0x800ac1000 rw- 28 28 2 1 CN-- df > 98756 0x800ac1000 0x800ad1000 rw- 16 16 2 1 CN-- df > 98756 0x800ad1000 0x800b91000 rw- 5 5 2 1 CN-- df > 98756 0x800b91000 0x800bd1000 rw- 2 2 2 1 CN-- df > 98756 0x800bd1000 0x800c11000 rw- 2 2 2 1 CN-- df > 98756 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df > 98756 0x800c51000 0x800c91000 rw- 1 1 2 1 CN-- df > 98756 0x800c91000 0x800cd1000 rw- 1 1 2 1 CN-- df > 98756 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df > 98756 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df > 98756 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df > 98756 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df > 98756 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph > > Regards > Steve From owner-freebsd-hackers@freebsd.org Tue Mar 28 08:48:24 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD5CED21AB0 for ; Tue, 28 Mar 2017 08:48:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B4E1DEF for ; Tue, 28 Mar 2017 08:48:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x236.google.com with SMTP id w11so77183202wrc.3 for ; Tue, 28 Mar 2017 01:48:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=Dldxy/rK/bu7c0wneu0gg+m/xmwKaAaiyWsRGyBigOY=; b=OuPbcJk+VLR97YJZRQXdjlo6Hj5W9u8ZkpC4yKxLFkd57AI04qf3z5UvcwQL72arW5 9tHTVWOq3DyuPQUPZTaGt/4LvDuiD1VX/Y5j3J5iqtIw3IaASP1ZtrKl9j7Vignomcux gy4M/Z7hdHA0Nh9TfrkNU0mW68+ZQ4xMHEXtUpBSQpQEBMMi8Ep6er2vhQLPV+L10O5p 6ZjqpM9v5jBRuUW1F9pabMK9mXCKwqQ5wuCD4tO8WaImLvuFjuX1F+UxCz03ks2M15Vw 6PuPwOYkZMoA/FNagtMb/eM2i/vVKY6Duyu8VmKmNGzGfOg8g4uqe5mac06EzfN+sWz3 Z2eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=Dldxy/rK/bu7c0wneu0gg+m/xmwKaAaiyWsRGyBigOY=; b=B60uAJzFnYMNPk/614shierSFYqjQR+vbVtdukVaxG5e2+ajkev55ZA/5YQIXopYTd t5tPbrURLp34JrskOAwZn6HU05q07x9GypOWfLIzUqC1qCTCtEgdLdQJnQzd+9v7e9p5 ZNLUP9zpad326Qi7eo8j98m6Woi8/UOmc+l2QqqvZFnwp9yvKD4gtEe7iu5aOwyAig8i KfqalduzxTUKtskggqGtvk4ptqIuYAEi1i1ouP6NAaWbDrNiKxX4XpT0b+SVhus/9Oxl NzboofVqElsDs7IRmTRnqHuFPtoIOqaOKv+paPWvRzNCqgR5xrVdtb6ovxp/SN/jFDf7 gQKw== X-Gm-Message-State: AFeK/H0isoavqL4NIuDXkZ5gvSPtYlIfondrhHvAw4OGPaPa/zF6aMn/zInBjlH3wdVW33yZ X-Received: by 10.28.105.8 with SMTP id e8mr13616363wmc.122.1490690902436; Tue, 28 Mar 2017 01:48:22 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id e16sm3911486wra.62.2017.03.28.01.48.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 01:48:21 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> <20170328083810.GR43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <5aa653ba-30e1-c9de-46ce-bad74d78c40c@multiplay.co.uk> Date: Tue, 28 Mar 2017 09:48:23 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170328083810.GR43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 08:48:24 -0000 On 28/03/2017 09:38, Konstantin Belousov wrote: > On Tue, Mar 28, 2017 at 09:23:24AM +0100, Steven Hartland wrote: >> As I stopped the panic before that I couldn't tell so I've re-run with >> some debug added just before the panic to capture the addresses of the >> workbuf structure that the issue was detected in, here goes (parent: >> 62620, child: 98756): >> >> workbuf: 0x800b51800 >> fatal error: workbuf is not empty >> workbuf: 0x800a72000 >> fatal error: workbuf is empty >> workbuf: 0x800a72000 >> fatal error: workbuf is not empty > I do not understand. Why do you show several addresses ? Wouldn't the > runtime panic after detecting the discrepancy, so there could be only one > address ? There are several goroutines (threads) running each detected an error, as I'm blocking the panic by entering a sleep in the faulting goroutine to enable the capture of procstat, other routines continue and detect an error too. To be clear this is not exclusive to the addition of the sleep, as I've seen this without any changes, as the GC is multi threaded. > >> procstat -v 62620 98756 >> PID START END PRT RES PRES REF SHD FLAG >> TP PATH >> 62620 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 62620 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 62620 0x951000 0x988000 rw- 31 0 1 0 C--- vn >> /root/golang/src/test5/test5 >> 62620 0x988000 0x9ab000 rw- 18 0 1 0 C--- df >> 62620 0x800951000 0x8009f1000 rw- 36 0 1 0 C--- df >> 62620 0x8009f1000 0x800ac1000 rw- 28 0 1 0 C--- df >> 62620 0x800ac1000 0x800ad1000 rw- 16 0 1 0 C--- df >> 62620 0x800ad1000 0x800b91000 rw- 5 0 1 0 C--- df >> 62620 0x800b91000 0x800bd1000 rw- 2 0 1 0 C--- df >> 62620 0x800bd1000 0x800c11000 rw- 2 0 1 0 C--- df >> 62620 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df >> 62620 0x800c51000 0x800c91000 rw- 1 0 1 0 C--- df >> 62620 0x800c91000 0x800cd1000 rw- 1 0 1 0 C--- df >> 62620 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 62620 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 62620 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 62620 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df >> 62620 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> 98756 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x951000 0x988000 rw- 31 0 2 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df >> 98756 0x800951000 0x8009f1000 rw- 36 36 2 1 CN-- df >> 98756 0x8009f1000 0x800ac1000 rw- 28 28 2 1 CN-- df >> 98756 0x800ac1000 0x800ad1000 rw- 16 16 2 1 CN-- df >> 98756 0x800ad1000 0x800b91000 rw- 5 5 2 1 CN-- df >> 98756 0x800b91000 0x800bd1000 rw- 2 2 2 1 CN-- df >> 98756 0x800bd1000 0x800c11000 rw- 2 2 2 1 CN-- df >> 98756 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df >> 98756 0x800c51000 0x800c91000 rw- 1 1 2 1 CN-- df >> 98756 0x800c91000 0x800cd1000 rw- 1 1 2 1 CN-- df >> 98756 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 98756 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 98756 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 98756 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df >> 98756 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> >> Regards >> Steve From owner-freebsd-hackers@freebsd.org Tue Mar 28 09:57:22 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5804D1DE2F for ; Tue, 28 Mar 2017 09:57:22 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22f.google.com (mail-wr0-x22f.google.com [IPv6:2a00:1450:400c:c0c::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 42134BEC for ; Tue, 28 Mar 2017 09:57:22 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22f.google.com with SMTP id w11so79546186wrc.3 for ; Tue, 28 Mar 2017 02:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=uNd0T4uKvQMJR3lJ8D8jmAZ61w5DedSuF6xGfSwa+eQ=; b=fAPWuEW9JEEOdlGAEjC4UBH9wcHmoywzFjM+9uy2lt8rCGWvx546rjnBltsgicYDeT viKFBF3GxrMcouqMdvbwCY7OnL0RVOjZA/JD48Q7Xz2XHx60ib0qoHXZVCB7Lsa0d2Ws gV7gec14G5dWTfzkNLstKqeU1X/QDAARHOuocAGkqBJ4iaE8RXiarOowEBu5/aV1sCkN RNUsRnqYdSgXgjsDgd4DAxZKJRDygr9HoEngL6vwTUsR5TU4fSKNY3AisR0viCS/iadQ XgmTVdpfJa62BjSkPFzMcAbALk2OX1IUcdjbNGA9EPqmi+Nyw/5fUEK4zxeezva6ngR+ dJWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=uNd0T4uKvQMJR3lJ8D8jmAZ61w5DedSuF6xGfSwa+eQ=; b=CWDWTCHe6IvMdogxLvEeOlS6nJp0ZCjZXkFakavVTJ5D+rLrZScb+nPqkxQDLZ6ItL lEEplWvP1so63JSbpQeclxdi1s0JvyzrH4loargnjFA+UYVCu37ffDIQv73JPpVKdJlJ UW7tl7NzaGA1Unoj0DS86/hMEh92iOzz/5d3yb++P2kJJy/hkXsdzB8L190DCX+lBIva 8nVlx+WgHncPOEQDYm9kVCBCHMDLApjQvDoDo86nEyCTpxRZqsDRTkd46KdS0sm7tlDs 0jd7htsguR1GN595Ajf3jF4ZHNYYKpXcLUh/ccMIIARzZKdv2+H7oBio6q6RAL9c3ecV J/ag== X-Gm-Message-State: AFeK/H3ANDlj91dxExxR1N45e7rqWedx7xyloNjhd/iIUZMxo+EC6vYXDsdQXjUjYn99Hqth X-Received: by 10.223.154.199 with SMTP id a65mr22481000wrc.78.1490695039504; Tue, 28 Mar 2017 02:57:19 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id c8sm4161132wrd.57.2017.03.28.02.57.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 02:57:18 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <955146f8-126f-b5ac-3001-7f643d434716@multiplay.co.uk> Date: Tue, 28 Mar 2017 10:57:20 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327164905.GN43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 09:57:22 -0000 On 27/03/2017 17:49, Konstantin Belousov wrote: > >>> Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? >> Yes it does, I don't currently have i386 machine to test with, I'm >> assuming testing i386 on amd64 kernel, would likely not have any effect. > Confirmed same issue on i386 as amd64 (tested using virtualbox). Regards Steve From owner-freebsd-hackers@freebsd.org Tue Mar 28 11:39:07 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 201DDD21E71 for ; Tue, 28 Mar 2017 11:39:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0B44288; Tue, 28 Mar 2017 11:39:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2SBcxe8069171 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Mar 2017 14:39:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2SBcxe8069171 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2SBcx2q069170; Tue, 28 Mar 2017 14:38:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Mar 2017 14:38:59 +0300 From: Konstantin Belousov To: Steven Hartland Cc: "K. Macy" , "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20170328113859.GS43712@kib.kiev.ua> References: <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> <20170328083810.GR43712@kib.kiev.ua> <5aa653ba-30e1-c9de-46ce-bad74d78c40c@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5aa653ba-30e1-c9de-46ce-bad74d78c40c@multiplay.co.uk> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 11:39:07 -0000 On Tue, Mar 28, 2017 at 09:48:23AM +0100, Steven Hartland wrote: > On 28/03/2017 09:38, Konstantin Belousov wrote: > > On Tue, Mar 28, 2017 at 09:23:24AM +0100, Steven Hartland wrote: > >> As I stopped the panic before that I couldn't tell so I've re-run with > >> some debug added just before the panic to capture the addresses of the > >> workbuf structure that the issue was detected in, here goes (parent: > >> 62620, child: 98756): > >> > >> workbuf: 0x800b51800 > >> fatal error: workbuf is not empty > >> workbuf: 0x800a72000 > >> fatal error: workbuf is empty > >> workbuf: 0x800a72000 > >> fatal error: workbuf is not empty > > I do not understand. Why do you show several addresses ? Wouldn't the > > runtime panic after detecting the discrepancy, so there could be only one > > address ? > There are several goroutines (threads) running each detected an error, > as I'm blocking the panic by entering a sleep in the faulting goroutine > to enable the capture of procstat, other routines continue and detect an > error too. Ok. So I tried to simulate the load with an isolated test. Code below is naive, but it should illustrate the idea. Parent allocates some number of private-mapped areas, then runs threads which write bytes into the areas. Simultaneously parent forks children which write distinct byte into the same anonymous memory. Parent checks that it cannot see a byte written by children. So far it did not tripped on my test machine. Feel free to play with it, if you have more insights what go runtime does, modify the code to simulate the failing test more accurately. /* $Id: cowfail.c,v 1.1 2017/03/28 11:29:58 kostik Exp kostik $ */ #include #include #include #include #include #include #include #include #include #include #include static char **areas; static int nareas, nchildren, children, nthreads; static size_t areasz; static const char parent_chars[] = "ab"; static const char child_char = 'c'; static int gen_idx(void) { return (random() % nareas); } static void fill_area(int idx, bool parent) { char *area; char f; area = areas[idx]; f = parent ? parent_chars[random() % sizeof(parent_chars)] : child_char; memset(area, f, areasz); } static void check_area(int idx) { char *area; size_t i; area = areas[idx]; for (i = 0; i < areasz; i++) { if (area[i] == child_char) errx(1, "corrupted area"); } } static void child(void) { int i, idx; for (i = 0; i < 100; i++) { idx = gen_idx(); fill_area(idx, false); } _exit(0); } static void * wthread(void *arg __unused) { for (;;) { fill_area(gen_idx(), true); check_area(gen_idx()); } return (NULL); } int main(void) { pthread_t thr; sigset_t sigs; pid_t pid; int error, i, status; nareas = 1024; nchildren = 8; nthreads = 4; areasz = 1024 * 1024; sigemptyset(&sigs); sigaddset(&sigs, SIGCHLD); error = sigprocmask(SIG_BLOCK, &sigs, NULL); if (error == -1) err(1, "sigprocmask"); areas = calloc(nareas, sizeof(char *)); if (areas == NULL) err(1, "calloc nareas"); for (i = 0; i < nareas; i++) { areas[i] = mmap(NULL, areasz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); if (areas[i] == MAP_FAILED) err(1, "mmap %d", i); } for (i = 0; i < nthreads; i++) { error = pthread_create(&thr, NULL, wthread, NULL); if (error != 0) errc(1, error, "pthread_create"); } for (;;) { if (children < nchildren) { pid = fork(); if (pid == -1) { err(1, "fork"); } else if (pid == 0) { child(); } else { children++; } } else { pid = waitpid(-1, &status, 0); if (pid == -1) { if (errno != EINTR) err(1, "waitpid"); } else { children--; } } } } From owner-freebsd-hackers@freebsd.org Tue Mar 28 19:22:46 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67B22D22192; Tue, 28 Mar 2017 19:22:46 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 32E31BE8; Tue, 28 Mar 2017 19:22:45 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.14.9/8.14.9) with ESMTP id v2SJ24bB036623; Tue, 28 Mar 2017 12:02:10 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) To: "FreeBSD Hackers" Cc: "FreeBSD CURRENT" From: "Chris H" Subject: is an r316100 world/kernel possible from a r314700 jail? Date: Tue, 28 Mar 2017 12:02:10 -0700 Content-Type: text/plain; charset=UTF-8; format=fixed MIME-Version: 1.0 Message-id: Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 19:22:46 -0000 While I *know* this at *least* risky business; I was wondering what the chances are that I can create a new world/kernel from my current custom world/kernel? I've built/installed world/kernel (12-CURRENT) tracking HEAD. Which is now at r314700. But was hoping I could test a copy of r316100 by building everything in an r314700 jail. Thanks for any thoughts you'd care to share on this! --Chris From owner-freebsd-hackers@freebsd.org Tue Mar 28 19:50:58 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26844D21153; Tue, 28 Mar 2017 19:50:58 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E34337E4; Tue, 28 Mar 2017 19:50:56 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.14.9/8.14.9) with ESMTP id v2SJpX88063952; Tue, 28 Mar 2017 12:51:39 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) To: "FreeBSD Hackers" Cc: "FreeBSD CURRENT" In-Reply-To: References: From: "Chris H" Subject: Re: is an r316100 world/kernel possible from a r314700 jail? Date: Tue, 28 Mar 2017 12:51:39 -0700 Content-Type: text/plain; charset=UTF-8; format=fixed MIME-Version: 1.0 Message-id: Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 19:50:58 -0000 On Tue, 28 Mar 2017 12:02:10 -0700 "Chris H" wrote > While I *know* this at *least* risky business; > I was wondering what the chances are that I can create > a new world/kernel from my current custom world/kernel? > I've built/installed world/kernel (12-CURRENT) tracking > HEAD. Which is now at r314700. But was hoping I could > test a copy of r316100 by building everything in an > r314700 jail. Sorry. That was a *real* stupid question. Sorry for the noise! I've got *way* too many irons in the fire today. Gonna need to slow down. Again, sorry. :-( --Chris From owner-freebsd-hackers@freebsd.org Tue Mar 28 22:50:40 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA829D23F82 for ; Tue, 28 Mar 2017 22:50:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59FB81350 for ; Tue, 28 Mar 2017 22:50:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x229.google.com with SMTP id u132so50889967wmg.0 for ; Tue, 28 Mar 2017 15:50:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=P18p0gQK25xTrHDARDdvr9TcHGtk46aqOnJujkDB/NQ=; b=jj1Nlg4VTFNu3xezRamK5bT4cTclJykrx3GgZLKAfcPw12/3ogRCepu360npYoTQ+5 LjBr0CoTrtruhH8OrKJsi+SRqtHhsm0IlsDFZLyhfAbVerUlL086BsgPh67yrBJopywL 2CWhpBx/hoSnru7wVmT87HMGMO4KctNtmZ/sOxM22EGbrPt07G+k0tT/Ifr8EjINO+yp TC4LpHhgD7c7NsyVo5HrLpo6Ow0C1AZvmH4bHBXm0SbF17MV/gBEP6hqtNkb7UyzpuaN UWBvMWeSdTJy2aiZoAxcEZSHrKrFJbTO2beAS7nIanhgSwJwEmgQRiw/Elessle+o7q+ qkhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=P18p0gQK25xTrHDARDdvr9TcHGtk46aqOnJujkDB/NQ=; b=LUvpr1kCC67h4Zpxbt6xcmTkHwuCd8qaHqX6sL0l8HjwrybBIK8XRFsoK1mOndRt53 N1bwszd1SycOEW8pMQ2eWYJXKHlJqIpHCgRZCzzreLCH411QP/zO6I+s9HpojN/Y1HXe Q7Pe+z8Eg1fbpTEumJWrtKAM3wuRYTNUrhg7Xk5Ud3jbty2q0yjSMCKYNXz2JDw/UPcz e78ZTPiTvLsu4rZYmYlmC1T2xuF/5Fn1Et7VqjXJ+WwVNwcX4xcTPJFcwck/L7+/Rr5v 7zdfakf7nWY/C2ReBiuk1GO5/CkMCIOHPEQ0pW48ZsTm2mG78gO4HNFWhqfRpCrw+AvQ nnZg== X-Gm-Message-State: AFeK/H30Omy03L/wwCjQF66Ucw9px8Uok7ZpTV6BIz9Rj9FMXHR3WejVhy74o+lU5nl3qc+c X-Received: by 10.28.0.136 with SMTP id 130mr16781120wma.126.1490741438475; Tue, 28 Mar 2017 15:50:38 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id y107sm6585640wrc.35.2017.03.28.15.50.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 15:50:37 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> <20170328083810.GR43712@kib.kiev.ua> <5aa653ba-30e1-c9de-46ce-bad74d78c40c@multiplay.co.uk> <20170328113859.GS43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <2f5952b5-104d-4e82-5ebe-8cb9caabedae@multiplay.co.uk> Date: Tue, 28 Mar 2017 23:50:37 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170328113859.GS43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 22:50:40 -0000 On 28/03/2017 12:38, Konstantin Belousov wrote: > On Tue, Mar 28, 2017 at 09:48:23AM +0100, Steven Hartland wrote: >> On 28/03/2017 09:38, Konstantin Belousov wrote: >>> On Tue, Mar 28, 2017 at 09:23:24AM +0100, Steven Hartland wrote: >>>> As I stopped the panic before that I couldn't tell so I've re-run with >>>> some debug added just before the panic to capture the addresses of the >>>> workbuf structure that the issue was detected in, here goes (parent: >>>> 62620, child: 98756): >>>> >>>> workbuf: 0x800b51800 >>>> fatal error: workbuf is not empty >>>> workbuf: 0x800a72000 >>>> fatal error: workbuf is empty >>>> workbuf: 0x800a72000 >>>> fatal error: workbuf is not empty >>> I do not understand. Why do you show several addresses ? Wouldn't the >>> runtime panic after detecting the discrepancy, so there could be only one >>> address ? >> There are several goroutines (threads) running each detected an error, >> as I'm blocking the panic by entering a sleep in the faulting goroutine >> to enable the capture of procstat, other routines continue and detect an >> error too. > Ok. > > So I tried to simulate the load with an isolated test. Code below is > naive, but it should illustrate the idea. Parent allocates some > number of private-mapped areas, then runs threads which write bytes into > the areas. Simultaneously parent forks children which write distinct > byte into the same anonymous memory. > > Parent checks that it cannot see a byte written by children. > > So far it did not tripped on my test machine. Feel free to play with it, > if you have more insights what go runtime does, modify the code to simulate > the failing test more accurately. I've updated to it to be more like the go, so single forking thread (non-main), ancillary threads mainly idle until triggered by forking thread to perform a check, and still no failure. What's curious is why I don't get the issue if either: 1. The machine has just a single core. 2. The work (GC) is moved after the child wait. Given the above I added some debug: func (b *workbuf) checknonempty() { if b.nobj == 0 { print("workbuf is empty: b: ", b, ", nobj: ", b.nobj, ", nobj2: ", b.nobj2, ", pushcnt: ", b.node.pushcnt, "\n") throw("workbuf is empty") } } func (b *workbuf) checkempty() { if b.nobj != 0 { print("workbuf is not empty: b: ", b, ", nobj: ", b.nobj, ", nobj2: ", b.nobj2, ", pushcnt: ", b.node.pushcnt, "\n") throw("workbuf is not empty") } } Here's the output: workbuf is not empty: b: 0x800c51000, nobj: 4, nobj2: -2, pushcnt: 104881 fatal error: workbuf is not empty Nothing strange, but now lets have a look using gdb after the parent has exited: (gdb) frame 8 #8 0x000000000041f1e8 in runtime.(*workbuf).checkempty (b=0x800c51000) at /usr/local/go/src/runtime/mgcwork.go:328 328 throw("workbuf is not empty") (gdb) print b $3 = (struct runtime.workbuf *) 0x800c51000 (gdb) print *b $4 = {runtime.workbufhdr = {node = {next = 0, pushcnt = 104881}, nobj = 0, nobj2 = -8},.... So after the error was printed the value for nobj was some how corrected, however nobj2 being -8 indicates the last call which altered nobj was func (w *gcWork) get() uintptr where as the -2 indicates it was a putfull which is very muddled up. I was curious what the child had at 0x800c51000 but couldn't persuade gdb to cast and output it as a struct runtime.workbuf. Regards Steve From owner-freebsd-hackers@freebsd.org Wed Mar 29 05:16:13 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3886D238E7; Wed, 29 Mar 2017 05:16:13 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F5E465082; Wed, 29 Mar 2017 05:16:13 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-wm0-f47.google.com with SMTP id o81so13467654wmb.1; Tue, 28 Mar 2017 22:16:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc:content-transfer-encoding; bh=cfaL7d0AYAdRMg9CVapyydtP2Zwlp2vaWMhd8ZJys94=; b=buLBmgJoeAkuGlAqx9pjeb6snCsZbgoxn/+bSJjciZSnDmD/xmSf9yDxAuetAtDbYa m0kOnM5txbQnysMYIhr9bUtsDTRnWxngH1CCIguyfStioZQBJMbvxnGpc5HMBRL0OMMk UTSHlJw8XyZ3czZGdRzErac210kw5HtN9HAbflOSCtYU8G+IeiJu6lUUznS4TcF6lPyx 8Hd9z12Kkf2WH2GA3ZFflco7u4SswOCves4cURVvgXNXX+j6x8RFVuooKzGhbJ15Wtf9 0kHW4/9YOhq1zv8fRKUSOAYSPSAgYA3U6B1b/9b9hI+sOyOm9lopHRwe7ru4IZzLhHcU /Y/g== X-Gm-Message-State: AFeK/H1YKNS49wxINlofs3NxADXnRss6ASrEyxLIADGjsQFidC8vNf2oN299L3l3bq7Abg== X-Received: by 10.28.15.12 with SMTP id 12mr17290810wmp.22.1490764565575; Tue, 28 Mar 2017 22:16:05 -0700 (PDT) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com. [74.125.82.45]) by smtp.gmail.com with ESMTPSA id h3sm7469138wrb.6.2017.03.28.22.16.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 22:16:05 -0700 (PDT) Received: by mail-wm0-f45.google.com with SMTP id u132so54631190wmg.0; Tue, 28 Mar 2017 22:16:04 -0700 (PDT) X-Received: by 10.28.232.21 with SMTP id f21mr12158120wmh.0.1490764564852; Tue, 28 Mar 2017 22:16:04 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.80.169.4 with HTTP; Tue, 28 Mar 2017 22:16:04 -0700 (PDT) In-Reply-To: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> From: Conrad Meyer Date: Tue, 28 Mar 2017 22:16:04 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Proposal for a design for signed kernel/modules/etc To: Eric McCorkle Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 05:16:14 -0000 Hi Eric, Your proposal seems reasonable to me. Others =E2=80=94 if you don't have time to read the full email, start readi= ng at "=3D=3D Proposal=3D=3D" for a summary of what is actually proposed. You can go back and read the earlier part of the email for some discussion of requirements and other options/context. Thanks, Conrad From owner-freebsd-hackers@freebsd.org Wed Mar 29 06:11:58 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39CB7D23319; Wed, 29 Mar 2017 06:11:58 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE9DD67442; Wed, 29 Mar 2017 06:11:57 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: by mail-qk0-x231.google.com with SMTP id r142so4651744qke.2; Tue, 28 Mar 2017 23:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=nTYPCuJs0lEF3GlEVhyQVX6O/kTafl4IKg4HomdkfXA=; b=sS6Mgx6fBeEdRIPtISGbe74uiAP/uswdxaKlffM3Wzuf9GfdzamHsxbAbGL04UzxQG uW8JTylaySDMZqOYGQL8eg2Uf3TsxDdQ3N4dwaecBOwgJL+/5M2hnVZOKwVbuYMDdhp/ VTHlds6IIK/b2cY/XRHxt6G4AngBroQE7a83zODjcNAKSldn7vUxBgYAeH+6EXiSI4co nU9wcgUQQsZjKXKbnI0J5YgDXbN5MFC5MjVG43l7Ot9Y2BjtH7OebMy50b+f2/LR+E8r X2MwXITdDqNiupevEvmmJdEzGOJhbLMP3TeAa1MQDlG2Y7m03210vZ4iule7RKQ6S+AY A9hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=nTYPCuJs0lEF3GlEVhyQVX6O/kTafl4IKg4HomdkfXA=; b=o6a+jZUTNumbuOpbA7dLmPaiVozGHWqAJa18IU8Pb52TqZCqTXy6HhvR5e4p6BO8fG l7sTz9/fYA/fI8MXsu1So2+DmvW5SIcWXwJF15qN5/+LI8UT8xTV20ObJdFmbPKLL6CC gJ7p61zUSWM6nY3DckaUWxlFc7uP7jclJIDPFJPyTt/XUt6Elxr9LihPAd7L98QyCNq1 SqBwaluaNBwW3lKr7JpMQpts+CMNozC82bvBr7drD3EdfMw5oXC9CpNkuSH5xmoHxklO cSHgbTfVGyNfP80Vkwo5rh003OtqHzHu0OTn8vLLf6Qrsu2VleANHz2XsnL/HNfUpcPX lRpw== X-Gm-Message-State: AFeK/H2ahNvW5+zxGgk5kFInrIAZpR81sF+Wn0yL9RtAY648re7Xc2xTipCi3/mQqjACAk1/a+hHvIeZEWEsEw== X-Received: by 10.55.47.4 with SMTP id v4mr28565282qkh.77.1490767916809; Tue, 28 Mar 2017 23:11:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.38.146 with HTTP; Tue, 28 Mar 2017 23:11:56 -0700 (PDT) From: Shrikanth Kamath Date: Tue, 28 Mar 2017 23:11:56 -0700 Message-ID: Subject: ldscript.arm and elf section ordering To: freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 06:11:58 -0000 Have a question around the order in which symtab/strtab/shstrtab are placed for the elf binaries. Working with two different versions of the LLVM toolchain here in Juniper, so when building FreeBSD kernel with llvm 3.5 toolchain for arm (toolchain is published from sources internally) the sections shstrtab/symtab/strtab have offsets that are in increasing order (this is skimmed output from readelf) [Nr] Name Type Addr Off Size ES Flg Lk Inf Al ... [44] .shstrtab STRTAB 00000000 ede7f2 000283 00 0 0 1 [45] .symtab SYMTAB 00000000 edf1d0 05fc40 10 46 15427 4 [46] .strtab STRTAB 00000000 f3ee10 0834a2 00 0 0 1 When building with llvm 3.7 though the offsets do not appear to be in the expected order [40] .shstrtab STRTAB 00000000 10cbdaa 000229 00 0 0 1 [41] .symtab SYMTAB 00000000 fd0520 079de0 10 42 22109 4 [42] .strtab STRTAB 00000000 104a300 081aaa 00 0 0 1 the .shstrtab offset appears to be beyond section ".strtab" and ".symtab" but in terms of section ordering .shstrtab appears before symtab and strtab. Due to some design constraint I need to investigate if we can fix the .shstrtab beyond .symtab and .strtab, query to hackers is if this is an advisable thing to do? And how should the ldscript.arm be used to reorder this. Thanks, -- Shrikanth R K From owner-freebsd-hackers@freebsd.org Wed Mar 29 12:16:18 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0870ED23101 for ; Wed, 29 Mar 2017 12:16:18 +0000 (UTC) (envelope-from roam@ringlet.net) Received: from nimbus.fccf.net (nimbus.fccf.net [77.77.144.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A90696926D for ; Wed, 29 Mar 2017 12:16:17 +0000 (UTC) (envelope-from roam@ringlet.net) Received: from office.storpool.com (unknown [46.233.30.128]) by nimbus.fccf.net (Postfix) with ESMTPSA id 61006297 for ; Wed, 29 Mar 2017 15:10:53 +0300 (EEST) Received: from roam (uid 1000) (envelope-from roam@ringlet.net) id 25403fc by office.storpool.com (DragonFly Mail Agent v0.11); Wed, 29 Mar 2017 15:10:52 +0300 Date: Wed, 29 Mar 2017 15:10:52 +0300 From: Peter Pentchev To: Eric McCorkle Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Subject: Re: Proposal for a design for signed kernel/modules/etc Message-ID: <20170329121052.l6e7ajvvq6yfltpt@office.storpool.com> Mail-Followup-To: Eric McCorkle , "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="2y6ant3anwcqndju" Content-Disposition: inline In-Reply-To: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> User-Agent: NeoMutt/20170113 (1.7.2) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 12:16:18 -0000 --2y6ant3anwcqndju Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 27, 2017 at 01:54:44PM -0400, Eric McCorkle wrote: > Hello everyone, >=20 > The following is a design proposal for signed kernel and kernel module > loading, both at boot- and runtime (with the possibility open for signed > executables and libraries if someone wanted to go that route). I'm > interested in feedback on the idea before I start actually writing code > for it. >=20 > =3D=3D Goals =3D=3D >=20 [snip] >=20 > =3D=3D Non-Goals =3D=3D >=20 [snip] >=20 > =3D=3D Existing Solution(s) =3D=3D >=20 [snip] > While functional, this design doesn't meet the goals I outlined: >=20 [snip] > * Finally, the gnupg signature format doesn't actually seem to be > documented anywhere, or at least not anywhere that doesn't require a lot > of digging... Erm, actually, the so-called "gnupg signature format", better known as "the OpenPGP signature format", is pretty well documented in RFC 4880. Note that this remark has no bearing on any of your other arguments, or on your work as a whole; I just wanted to clarify this particular point :) G'luck, Peter --=20 Peter Pentchev roam@ringlet.net roam@FreeBSD.org pp@storpool.com PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 --2y6ant3anwcqndju Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEELuenpRf8EkzxFcNUZR7vsCUn3xMFAljbpEwACgkQZR7vsCUn 3xOT7w/+Ico4GO6e9BFic2UJs4n3cgyo/tzEwbV+JZ75w5uhPHeBmxsQo6q49Bbu JaCas1w7mPpDIiKK5oBLUTDVSEsHyDfBfBQn6yY7qax/pehi8EAAmxy6cLBj5LNL 4BJUrUbGlJVWe4Y/kxFiCxUhkDYTHzJTv7G2BsQeH/xGDhAnXtcNs7TSpxuTrUK8 jimOKLWNmxUPLuxTPIDuVmzV6nXFc7wrrdiqWOyaxOG7t16auuaou0OHIs0PBxND ZIXh5OBXQWhIW1DhQBTx5Anmi6oihOzeQSw1Ppt7OMoIbpZVwX+y8VYW6NFDCa9H 1xuLdqsxnMGuUsZw0QAgEfnEU7oFXnmhjcGLFL0AOabk2vP820bLhWNYefBDXlFw WYW677hrpCnNMT8SphUE4uHURJ93RnSudwRQVpkb6pdRAUr2iIOx3R6gsqJhs7gU HiUzoiiMhcizKyXRjrYLDL131HD1fCXwk967I7ggcDccfJ3v0FWblacMYsfRy8mT yS6234vL10tRkFMifQ65s5EVzsfiTUxYJubJhnGBqDWhiWrpytqQogRtgYqBbp3t zimxG8/jZ/h6eM+BeQuEdz9qqCwCa+Y+fUhV3SA7UEhpJKW3fXkKB6sFTagVjTyD 1pfQ7iUySFdnDDmMvOd/SJFU5jYfWRPUMy38iXD90T4jUSYtd54= =aH/a -----END PGP SIGNATURE----- --2y6ant3anwcqndju-- From owner-freebsd-hackers@freebsd.org Wed Mar 29 14:02:05 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E16D7D23BAE; Wed, 29 Mar 2017 14:02:05 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id B78523D35; Wed, 29 Mar 2017 14:02:05 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f] (unknown [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 1AE4116AF; Wed, 29 Mar 2017 14:02:05 +0000 (UTC) Subject: Re: Proposal for a design for signed kernel/modules/etc To: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170329121052.l6e7ajvvq6yfltpt@office.storpool.com> From: Eric McCorkle Message-ID: <5f2e1bf1-947d-3e2d-b7e4-034f8f1af3e9@metricspace.net> Date: Wed, 29 Mar 2017 10:02:02 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170329121052.l6e7ajvvq6yfltpt@office.storpool.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="g9p1eA8eXHkgsXgRu9Obp2UkogJhfcsTg" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 14:02:06 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --g9p1eA8eXHkgsXgRu9Obp2UkogJhfcsTg Content-Type: multipart/mixed; boundary="SOUBPUWIbUil86cK1NdLg2cFwNoTC56vo"; protected-headers="v1" From: Eric McCorkle To: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Message-ID: <5f2e1bf1-947d-3e2d-b7e4-034f8f1af3e9@metricspace.net> Subject: Re: Proposal for a design for signed kernel/modules/etc References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170329121052.l6e7ajvvq6yfltpt@office.storpool.com> In-Reply-To: <20170329121052.l6e7ajvvq6yfltpt@office.storpool.com> --SOUBPUWIbUil86cK1NdLg2cFwNoTC56vo Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/29/2017 08:10, Peter Pentchev wrote: >=20 > Erm, actually, the so-called "gnupg signature format", better known as > "the OpenPGP signature format", is pretty well documented in RFC 4880. > Note that this remark has no bearing on any of your other arguments, or= > on your work as a whole; I just wanted to clarify this particular point= :) Noted, though I think I'd prefer a format that's directly supported by OpenSSL. > G'luck, > Peter >=20 --SOUBPUWIbUil86cK1NdLg2cFwNoTC56vo-- --g9p1eA8eXHkgsXgRu9Obp2UkogJhfcsTg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEzzhiNNveVG6nWjcH1w0wQIFco2cFAljbvloACgkQ1w0wQIFc o2e1jw//Vo9ldXF8hVgQKTxCcHneVlCwsgqh0CacUs2GRj4aufoSSlqj9cQgVoEv gyTFd4sgx5MZ/7l1XRN1lBFsT0ulTb+ey+43JLFdRMfPaMZeID4+m7R2sCmIDK3N 2mYsBC74tZ3OEc0cmHQNQkLgccG+0/XF7qtdafJVulM85+mH7jefvVyZx7mCUX2r W5L9CW4QL0+SusRfzLZ9TZ8VsD/Kiw0rWtiV77/7VxAVRaDLPTR8mSRwM81Du4gR SE6eLROJdLmxOWNbm7Q7/pFYlZ0OCLUGWTic+UsU3lSXlJBg2O26PA5FelZ9PaJh T2g3ADBGgSuvZGFlCAUj2MgP2IGaCJk+ZTCaHIsPiSjpD0T35G126nMPTkzFyFlz DOB13wDwcjbW7w9adGSxf3qkQ46xf2vD4PtrCDf0VhkxtaqjbdXRkk8BXxD/YRgH oSCkHl39Z2l4s4CdMt43xnwaJuflGc8vuDEkcpbSdpwHir1h61sqmRn3F/XOzBDC Mu4oUhjpgviUnUSwDI9FkE+/ba2OnZbDi0/V2ktUzOm2ObyuZ6IJFXrdntrr+/J8 aCfEHt+gH/xca3z/Nf/rmx+3ewTLpFpfKq1T5lrhX5w8YZOyUyoudtjN31wLEP6M 2P9Pc0WKVtBmXz3nKj5Bhn3a4uqJJTHJfwVYqDrokiUfTWmOlR0= =pNNj -----END PGP SIGNATURE----- --g9p1eA8eXHkgsXgRu9Obp2UkogJhfcsTg-- From owner-freebsd-hackers@freebsd.org Wed Mar 29 20:00:43 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67FD1D24F5F for ; Wed, 29 Mar 2017 20:00:43 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [199.48.133.146]) by mx1.freebsd.org (Postfix) with ESMTP id 50EE4C49 for ; Wed, 29 Mar 2017 20:00:42 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id C748D5646B for ; Wed, 29 Mar 2017 15:00:36 -0500 (CDT) From: Eric van Gyzen Subject: One Priority Per Run Queue To: freebsd-hackers@freebsd.org Message-ID: <1aafd6a2-828c-06f5-bdac-e4c953a403b5@FreeBSD.org> Date: Wed, 29 Mar 2017 15:00:36 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 20:00:43 -0000 The FreeBSD schedulers assign four priorities to each run queue, making those priorities effectively equal. This breaks POSIX real-time priorities. Applications that use real-time scheduling use sched_get_priority_min() and sched_get_priority_max() [0] to determine the available range of priorities, and then use simple arithmetic to assign relatively higher or lower priorities. If an application configures two threads with priorities MAX and MAX-1 (for example), POSIX says the thread at priority MAX must be chosen if it is runnable. Since our implementation puts these two priorities in the same run queue, it may choose either thread, so it does not conform. The above functions currently return 0 and 31, respectively. One solution would change max() to return 7 and change other code to translate the 8 POSIX values into the 32 FreeBSD values. However, this would also not conform, because "conforming implementations shall provide a priority range of at least 32 priorities for this policy." [1] I propose that we assign one priority per run queue: https://reviews.freebsd.org/D10188 This would conform to POSIX. On a certain commercial block storage product, this change made no difference in performance. Benchmarks of buildworld on two different machines actually showed a tiny improvement in performance. [2] Please test the above change, especially if you have an interesting workload that might be sensitive to scheduler behavior. If you already know this change would cause problems, please point me toward the details. Assigning 4 priorities per run queue also caused a recent portability issue in ZFS, although that was fixed by r314058. Eric [0] http://pubs.opengroup.org/onlinepubs/9699919799/functions/sched_get_priority_max.html [1] http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_08_04_01 [2] http://www.vangyzen.net/FreeBSD/1ppq/1ppq.html From owner-freebsd-hackers@freebsd.org Wed Mar 29 21:18:54 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE549D249DE for ; Wed, 29 Mar 2017 21:18:54 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB76C16D for ; Wed, 29 Mar 2017 21:18:54 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x230.google.com with SMTP id f84so7419100ioj.0 for ; Wed, 29 Mar 2017 14:18:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=pqw/Cp3T0QKPN+fkxAEr5wnv6k46nQx4JcivkkTXrcA=; b=kvOOVgILFs3z0hhKCFYD1mD8Eai3iviXxAMHW8BNJKYNc6B5WM4c0Gw58Iaws1o710 XcbjX5S6wTFFkQWj3ffnmDdn4UdGBDCK1kuJUWL2hxIUTkQtJbYn0Cq3JnICo8O/z2us iINWfG0m8aHyNA2sYEq8A7WPav9YAotF4FPsRal6w2UZkzStmdzXO9zUN9bmWxzmhLC2 N3K0+Ql7tpBJ0XkSLrzlu9a5RK3d9Th3qGp2YMe5qWewfnu6Pud8Vc29miA3em0YiE5l 3efTPB7bVBr+twQxDWxCrXrEz56T8KT9mBqxomUJ6cuM5DeZszHSEDx7Py5cztUPsm3k NWlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=pqw/Cp3T0QKPN+fkxAEr5wnv6k46nQx4JcivkkTXrcA=; b=Df9oUnBiUgeAkGIsD9RzYjulA1x1WbPi6L7YAMkFSfxg/SijifXVBvyRzUGnUC9S4t 4/zpqwezlLTprUHpDtBu8UkVgD1YTAQjSo92sSLZiy4SovkoSPzA62zV7FGrfz063cdP 3Yrz17eCVG8J28bBTQkBVVGz4zqclyoZk4rbOjQNmfYijefjuiIn9xBtGtvkRov4JHG8 RpGmo5mRy5bvRdBQ5jQJPSKGU7AP2pfd/zeuyaZmpPFdJ4rr5ZKGLWAoDxLVXcOrLJ2M JAIu2Rn8qcooB8m6c/ce4r0nHIESExDD2HL7UlEF69hQnQxXGsBySjQYSI8Qki9aUhkD MF3g== X-Gm-Message-State: AFeK/H2EoBD398mCR38qs5mxQiqXgSEWQ+hcZ9xskkpEWEwrn7cVzmYTBmkR4QHNyCuIRyuMZ46jLAxKv9rT3A== X-Received: by 10.107.198.193 with SMTP id w184mr3069265iof.19.1490822333804; Wed, 29 Mar 2017 14:18:53 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.146.24 with HTTP; Wed, 29 Mar 2017 14:18:53 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:a045:c633:4156:d07c] In-Reply-To: <1aafd6a2-828c-06f5-bdac-e4c953a403b5@FreeBSD.org> References: <1aafd6a2-828c-06f5-bdac-e4c953a403b5@FreeBSD.org> From: Warner Losh Date: Wed, 29 Mar 2017 15:18:53 -0600 X-Google-Sender-Auth: mnnwpx_dfnUCxqMQ5qDtAA5wC_0 Message-ID: Subject: Re: One Priority Per Run Queue To: Eric van Gyzen Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2017 21:18:55 -0000 On Wed, Mar 29, 2017 at 2:00 PM, Eric van Gyzen wrote: > The FreeBSD schedulers assign four priorities to each run queue, making > those priorities effectively equal. This breaks POSIX real-time priorities. > > Applications that use real-time scheduling use sched_get_priority_min() > and sched_get_priority_max() [0] to determine the available range of > priorities, and then use simple arithmetic to assign relatively higher > or lower priorities. If an application configures two threads with > priorities MAX and MAX-1 (for example), POSIX says the thread at > priority MAX must be chosen if it is runnable. Since our implementation > puts these two priorities in the same run queue, it may choose either > thread, so it does not conform. > > The above functions currently return 0 and 31, respectively. One > solution would change max() to return 7 and change other code to > translate the 8 POSIX values into the 32 FreeBSD values. However, this > would also not conform, because "conforming implementations shall > provide a priority range of at least 32 priorities for this policy." [1] > > I propose that we assign one priority per run queue: > > https://reviews.freebsd.org/D10188 > > This would conform to POSIX. On a certain commercial block storage > product, this change made no difference in performance. Benchmarks of > buildworld on two different machines actually showed a tiny improvement > in performance. [2] > > Please test the above change, especially if you have an interesting > workload that might be sensitive to scheduler behavior. If you already > know this change would cause problems, please point me toward the details. > > Assigning 4 priorities per run queue also caused a recent portability > issue in ZFS, although that was fixed by r314058. How does this scheme prevent starvation of low priority processes? Or rather, how will this change after this change. Warner > [0] > http://pubs.opengroup.org/onlinepubs/9699919799/functions/sched_get_priority_max.html > > [1] > http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_08_04_01 > > [2] http://www.vangyzen.net/FreeBSD/1ppq/1ppq.html > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Thu Mar 30 02:22:46 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31E7BD25425; Thu, 30 Mar 2017 02:22:46 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id DF200BB8; Thu, 30 Mar 2017 02:22:45 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f] (unknown [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id F19FA18D6; Thu, 30 Mar 2017 02:22:44 +0000 (UTC) Subject: Re: Proposal for a design for signed kernel/modules/etc To: Shawn Webb References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170327183735.uokjhjaafkawc2id@mutt-hbsd> Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org From: Eric McCorkle Message-ID: Date: Wed, 29 Mar 2017 22:22:41 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327183735.uokjhjaafkawc2id@mutt-hbsd> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="o2TtOFM9E1kLqGqmdpTQ5s2xSHI3daBLW" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Mar 2017 02:22:46 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --o2TtOFM9E1kLqGqmdpTQ5s2xSHI3daBLW Content-Type: multipart/mixed; boundary="Di4UCV0TwwPjjxxo7nhqsEfixbrNrXmGj"; protected-headers="v1" From: Eric McCorkle To: Shawn Webb Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Message-ID: Subject: Re: Proposal for a design for signed kernel/modules/etc References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170327183735.uokjhjaafkawc2id@mutt-hbsd> In-Reply-To: <20170327183735.uokjhjaafkawc2id@mutt-hbsd> --Di4UCV0TwwPjjxxo7nhqsEfixbrNrXmGj Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/27/2017 14:37, Shawn Webb wrote: > The only other major thing to discuss is supporting public key chaining= =2E > Ideally, digital signature support should also support chaining multipl= e > keys (similar to X.509 PKI). If the accepted solution supported cert > chaining, then the solution would be more modular. I don't want to go > down the route of the SSL/TLS PKI mess, but supporting chaining is a > must in some enterprise environments. >=20 > If we were to support chaining, we could even stuff the pubkey half of > the key material into another ELF section, so that if a key becomes > compromised, the old pubkey can be revoked from the trust store and a > new binary can be generated with new key material. The trusted root > doesn't need to be cycled as often. HardenedBSD, for example, > distributes the pubkey that corresponds to the signing privkey inside > the update tarball for binary updates[1]. This allows us to change key > material often if desired without the user even noticing. I've done more research and design work in this area, to the point where I think I can talk about specific formats. I've also done some really preliminary work on the signelf utility (just sketching out the various use cases and marking the API calls I'll need, no real implementation yet). Here's what I've got: =3D=3D Use Cases =3D=3D The command-line utility ought to be able to verify signed ELF files too (so people can sanity check things). There's basically three cases that emerge for signing: 1) Vendor key + ephemeral key: I point you at the vendor key-pair (the public key for which is presumably compiled into the loader/kernel), you generate an ephemeral key-pair. You then sign the ELF with the ephemeral key-pair, sign the ephemeral public key with the vendor key-pair and include it in the ELF as well, then securely delete the ephemeral private key. 2) Direct signing with an application key: I point you at an application key-pair, which is signed by a vendor key somewhere (which is presumably compiled into loader/kernel). You sign the ELF with the application key-pair and include the signed application public key in the ELF as well. (Case (1) is basically the same as this one, except we generate and sign a one-time application key) 3) Direct signing without a vendor key: I point you at a key, which you use to sign the ELF; however, you don't include the public key in the ELF (it's presumably compiled into loader/kernel). On the verification side, you have two cases: A) ELF with signed application key: You have a set of vendor public keys. The ELF has a signed application public key and a file signature produced with said application public key. You check that the file signature is valid using the application public key, then check that the application key signature is valid using the vendor public keys. B) Direct-signed ELF: You have a set of vendor public keys. The ELF has a file signature produced by one of them (presumably). You check that the file signature is valid using the vendor public keys. Cases (1) and (A) should be the default behavior on both sides, and the signelf utility ought to be able to batch-sign a whole bunch of files with one ephemeral key. Case (2) is included in anticipation of a situation where it's not feasible to do all the signing in one fell swoop. Cases (3) and (B) are simplified behavior for when a vendor key isn't needed (ex. some kind of all-in-one embedded system image). =3D=3D Formats =3D=3D There are two formats (or rather, families thereof): * OpenSSL- formats based around ASN.1 DER and PEM, PKCS#8 format for private keys (which supports both encrypted and unencrypted storage), X509 for public keys, PKCS#7 or its successor, CMS for signatures. * OpenPGP (GnuPG) formats OpenSSL is directly supported by base, and is arguably more widely-used (it's the foundation for SSL/TLS). It also provides direct support for cert-chaining as I discussed in the use case analysis (in fact, it can go way beyond that). GRUB needs public keys to be in GnuPG format. On the other hand, the gnutls library handles X509 and PKCS#7. There's also some interesting possibilities with vendor keys and PGP's web-of-trust functionality. Fortunately, the monkeysphere project (port security/monkeysphere) comes with utilities for converting between OpenSSL certs/keys and GnuPG keyring databases. With this in play, it just makes sense to use the formats that are directly supported by base. =3D=3D Specifics =3D=3D So, here's what I'm proposing WRT the gritty format details: * The command-line tool will expect to see public keys in X509 format and private keys in PKCS#8 format, both with PEM encoding. * The vendor keys will be stored in a standard location (say, /etc/keys or something), and there would be a build utility that converts the public keys into a C file containing the data, so it can be baked directly into loader and kernel. * A signed ELF will definitely contain a .sign section containing a single detached signature in PKCS#7 format with DER encoding. * Signatures are computed by hashing the contents of the entire file minus the .sign and .cert sections. (Crypto hashes effectively consume a byte stream, so this is pretty straightforward to do) * A signed ELF may also contain a .cert section, which contains a single X509 certificate. In use case (1), this would contain the ephemeral public key. In use case (2), this would contain the application public key. In use case (3), this section would be omitted. You could presumably include an entire cert-chain here, but I think that's probably overkill. An alternative to the .cert section would be to have the ability to point kernel/loader at an X509 cert and pull the whole thing in as long as there's a root signature coming from a valid vendor key. I'm a bit split as to which one is better, though I seem to like the non-.cert solution the more I think about it... --Di4UCV0TwwPjjxxo7nhqsEfixbrNrXmGj-- --o2TtOFM9E1kLqGqmdpTQ5s2xSHI3daBLW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEzzhiNNveVG6nWjcH1w0wQIFco2cFAljca/EACgkQ1w0wQIFc o2dBtw/+M8Z/03eZqrmkN0WElSirvQTLU/orKe4t/XPJ+Rv7kqI3KjRhQEb76+0T 1KNJJrnhmiRiu9CEy8kVbatUQ75Hz5VmB4fbl4CeW5GQERSgaUp2ErPXT6Y4dENN dBc+yT+Sf0i1csN2lsfIVxD15bC+VweSuBDdsWHeEk7IjMBBNKaJ3TNxsGLTDV9f zgyiXLmwqgUXiR8l8dM3WavbwCdy0O9u/aNTS7gz+cFaeIRyjZaO7+041DSFFdjv ObymfvWCVEXmkIfFmsmtXCKCQSjJhyMmiqFYSKL2etM+gE0tdl9xHhjkviTXT7ov GlZCzxocLG1vMhzEfj5zEWOqHrQuu1NKmtSXv9yyaLGIHiTPSlro9bgDAlTlaThC vte3SRYNcFgazGTez2F8oqW87MrzsHzBA+hoeAbSyYiOO79e2EvgfmHsDxn/SvYl XGUHPPD+5OFek1wC28e/ofO3d99AMICZZzd5xc5Zif+JdAmSX6JeKk5HtqsXhk1P cV4s2ZF07PX8gHfcQtJJGz7TqJKZGdiV3iXZOrMaKQ/hmJ1R602M5pyMKH27sP8p e78IhlWw1tREbP0Koqk5Lrws5yFWyxaOr8TZobfhPYW3ZgvCAEbVVsFykOp3VXyu SLXSHiMInY7v55ih0johhh5YpXvCPj6xcYK5sW9HK5WbRoLu1/o= =qvz6 -----END PGP SIGNATURE----- --o2TtOFM9E1kLqGqmdpTQ5s2xSHI3daBLW-- From owner-freebsd-hackers@freebsd.org Thu Mar 30 03:25:34 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98986D17918; Thu, 30 Mar 2017 03:25:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0ACF769C; Thu, 30 Mar 2017 03:25:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v2U3PSAv094977 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 30 Mar 2017 06:25:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v2U3PSAv094977 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v2U3PSPq094976; Thu, 30 Mar 2017 06:25:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 30 Mar 2017 06:25:28 +0300 From: Konstantin Belousov To: Eric McCorkle Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Subject: Re: Proposal for a design for signed kernel/modules/etc Message-ID: <20170330032528.GT43712@kib.kiev.ua> References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> User-Agent: Mutt/1.8.0 (2017-02-23) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Mar 2017 03:25:34 -0000 On Mon, Mar 27, 2017 at 01:54:44PM -0400, Eric McCorkle wrote: > As background, the ELF file format has a number of different section > types, only some of which comprise the program/library/module's runtime > state. The ELF specification and tools provide some "standard" sections > with defined meanings, but nothing stops anyone from adding their own > sections. The ELF file format is quite flexible, and it is not > difficult to add custom metadata to an ELF file. [0] > > In this proposal, cryptographic signatures would be stored in a > .signature (or .sig) section. This section would contain an array of > signature constructs: one for each loadable segment in the ELF file. > Signatures are computed for the contents of the segment's file data (ie. > the data from p_offset to p_filesz, for the corresponding program header > entry) along with all data from its program header entry except for > p_offset and p_filesz. This scheme allows the actual data to be moved > around in the file, so long as it (or the relevant program header data) > isn't modified. First, you mix or do not understand a difference between section and segment. Second, this scheme allows to add loadable segments after signing. Third, most important, it has zero chances of working for amd64 modules. From owner-freebsd-hackers@freebsd.org Thu Mar 30 03:28:46 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3FBDBD17C11; Thu, 30 Mar 2017 03:28:46 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id 15619A26; Thu, 30 Mar 2017 03:28:46 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f] (unknown [IPv6:2001:470:1f11:617:3210:b3ff:fe77:ca3f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 1BD361911; Thu, 30 Mar 2017 03:28:45 +0000 (UTC) Subject: Re: Proposal for a design for signed kernel/modules/etc To: Konstantin Belousov References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170330032528.GT43712@kib.kiev.ua> Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org From: Eric McCorkle Message-ID: <164dec90-bfa3-4446-2ecb-49b257188d42@metricspace.net> Date: Wed, 29 Mar 2017 23:28:31 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170330032528.GT43712@kib.kiev.ua> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="02EUP2gS58WSV5tTurcbE566WE2PgVkSs" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Mar 2017 03:28:46 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --02EUP2gS58WSV5tTurcbE566WE2PgVkSs Content-Type: multipart/mixed; boundary="R4DT45iS3jdpLXk254IxS0UoUD1adVm26"; protected-headers="v1" From: Eric McCorkle To: Konstantin Belousov Cc: "freebsd-hackers@freebsd.org" , freebsd-security@freebsd.org Message-ID: <164dec90-bfa3-4446-2ecb-49b257188d42@metricspace.net> Subject: Re: Proposal for a design for signed kernel/modules/etc References: <6f6b47ed-84e0-e4c0-9df5-350620cff45b@metricspace.net> <20170330032528.GT43712@kib.kiev.ua> In-Reply-To: <20170330032528.GT43712@kib.kiev.ua> --R4DT45iS3jdpLXk254IxS0UoUD1adVm26 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/29/2017 23:25, Konstantin Belousov wrote: > First, you mix or do not understand a difference between section and se= gment. > Second, this scheme allows to add loadable segments after signing. > Third, most important, it has zero chances of working for amd64 modules= =2E That design has been abandoned in later discussions. --R4DT45iS3jdpLXk254IxS0UoUD1adVm26-- --02EUP2gS58WSV5tTurcbE566WE2PgVkSs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEzzhiNNveVG6nWjcH1w0wQIFco2cFAljce18ACgkQ1w0wQIFc o2f9kw/5AXtlmgobBmv1OUs4fhHn6i4+2PFV/nJey5twu3kyuJQZdUwl4TIudVy3 +ycN6lds4inoMfd+vx3sBCDW9rlN7L9/Ovuq6fYy3DBhX6k8GncCcgHyLpu9Wwxa U3IRW+mRUzo6feOnxl//0h6gqJzHWAIoxTlB49Lyh9gbwZWdVd+Mammj/i5FDgrp iaC57Zz1G/zlm0Ry4No7kloMUoJJXOPy1+ZBnM2VlGpA94hcJVCy5tnbQbMATQd9 6sJwGyW3ofOkr4b8/jzjK7N2wPHr/0JRZufMrxxarKqz8eRS6xtPf7ZNLV83oVxc rt/mPFiH+ZcWDtek1RlNNKUsvAmMzdy18G7QF0n8knI0YsW4Iv1dJts6TodRfhOm pDk0Iau+7bIe0LnALN82N2lMH9zwUE+iykYQC3SttooNRhYGs3lU4/MN7bt3Z3EB ErT5tadl6y8aRgMsO+1M1xAdF+AsrQ5WX31Op+2kCS3/PtDE3G2uP1xh2KJXYjPN s+WyU0n+THlqN2LIw4cESrnlEZUcVmqtjGcTjyQYbYNw8w6rZOSAECGZ9YOi+A78 FXJijg3HwcZ/Uv+KrnjpWTnnXYy8YPDcmq0zankjGwDEmUtfsalpqm2VpCTyjct9 ucgEPDX97GQTt0j3ZDRhSmPOzbYRmAAa0i9j+1BaNeRWbBH6SEk= =dYAg -----END PGP SIGNATURE----- --02EUP2gS58WSV5tTurcbE566WE2PgVkSs-- From owner-freebsd-hackers@freebsd.org Thu Mar 30 19:26:17 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29772D26670 for ; Thu, 30 Mar 2017 19:26:17 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from smtpq2.tb.ukmail.iss.as9143.net (smtpq2.tb.ukmail.iss.as9143.net [212.54.57.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DF7ED2DB for ; Thu, 30 Mar 2017 19:26:16 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from [212.54.57.81] (helo=smtp2.tb.ukmail.iss.as9143.net) by smtpq2.tb.ukmail.iss.as9143.net with esmtp (Exim 4.86_2) (envelope-from ) id 1cteuS-0006Oo-0O for freebsd-hackers@freebsd.org; Thu, 30 Mar 2017 20:35:00 +0200 Received: from oxbe11.tb.ukmail.iss.as9143.net ([172.25.160.142]) by smtp2.tb.ukmail.iss.as9143.net with bizsmtp id 2Jaz1v00L34e2W401Jazby; Thu, 30 Mar 2017 20:34:59 +0200 X-SourceIP: 172.25.160.142 X-Authenticated-User: j.deboynepollard-newsgroups@ntlworld.com Date: Thu, 30 Mar 2017 19:34:59 +0100 (BST) From: Jonathan de Boyne Pollard Reply-To: Jonathan de Boyne Pollard To: Debian users , FreeBSD Hackers , supervision@list.skarnet.org Message-ID: <736737774.3548811.1490898899979.JavaMail.open-xchange@oxbe11.tb.ukmail.iss.as9143.net> In-Reply-To: References: Subject: djbwares version 5 MIME-Version: 1.0 X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v7.6.2-Rev60 X-Originating-IP: 185.28.212.96 X-Originating-Client: com.openexchange.ox.gui.dhtml Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Mar 2017 19:26:17 -0000 djbwares is now at version 5. * http://jdebp.eu./Softwares/djbwares/ * http://jdebp.info./Softwares/djbwares/ This contains some long-overdue changes: ip6.int has been replaced by ip6.arpa in tinydns-data and dnscache, and rblsmtpd no longer falls back to using an RBL that has been defunct for many years. It also contains some additions: some UCSPI-SSL capability, a new gopherd UCSPI server to go alongside httpd and ftpd in publicfile, and most of the previously missing manual pages (including a few for commands which had no manuals in the original toolsets). There are no longer any placeholder manual pages for the "man" command. There are still a few manual pages that are only present in roff form, though. You can see gopherd in action: * gopher://jdebp.info./1/Repository/ From owner-freebsd-hackers@freebsd.org Thu Mar 30 19:26:29 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEE4AD266A2 for ; Thu, 30 Mar 2017 19:26:29 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7D5F0389 for ; Thu, 30 Mar 2017 19:26:28 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 1F287BE8 for ; Thu, 30 Mar 2017 20:19:36 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1490901576; bh=OE9Ln04OQyvZ8nBUj4kShoIM0kEDU8rnZEVWZ51VC4s=; h=From:To:Subject:Date; b=eTdS2pZZjK3FIn7uGMHhXfOrc5i10DFfVB+KxoVHz1LtVLGhOHESSjE4n9C6pNPaq pYNlDx6iiVBuTyaIi7GCrE75B9hBWKQAkhjYbkJTgjsv7nUhcUlU8ivbBbTc3ZxSAs y3FDJWVBcwhG93e5vM5YZ+lhz8x4xid3q4FatbbY= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Thu, 30 Mar 2017 20:19:35 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Thu, 30 Mar 2017 20:19:35 +0100 From: Matt Churchyard To: "freebsd-hackers@freebsd.org" Subject: Sendmail conf files Thread-Topic: Sendmail conf files Thread-Index: AQHSqYhJA05PurJ9FUaUzW3xtZ+p6A== Date: Thu, 30 Mar 2017 19:19:34 +0000 Message-ID: <481fef207e2e457cb4f8a689d0ce4373@SERVER.ad.usd-group.com> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [89.145.233.102] MIME-Version: 1.0 X-Mailman-Approved-At: Thu, 30 Mar 2017 19:56:14 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Mar 2017 19:26:29 -0000 Hello, Not sure if this is the right list for this.. For me the most awkward part of updating a system using freebsd-update is w= hen it comes to merging files. The most common files that pop up seem to be= /etc/mail/sendmail.cf & /etc/mail/submit.cf, because these are included in= the base distribution and almost certainly change if you actually use Send= mail. In most cases this sort of issue has been solved by providing "default file= s" such as /etc/defaults/rc.conf, and letting the user override this with f= iles they create themselves. Obviously there is a concerted effort to make = sure users don't have to edit base files where possible so that they don't = get these sort of issues. For Sendmail, would it not make sense to remove these 2 .cf files from base= and update the sendmail rc.d script to run 'make install' in /etc/mail if = they don't exist? It may also be nice if the freebsd.* files were stored so= mewhere else such as /usr/share/sendmail, as these just causes confusion ab= out which files are actually used if you're new to it. Personally I'm on the side that would rather have Sendmail removed entirely= and replaced with a simple smtp submission daemon/lda but I think that dis= cussion has already been had. - Matt Churchyard From owner-freebsd-hackers@freebsd.org Fri Mar 31 03:54:49 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9C65D26661 for ; Fri, 31 Mar 2017 03:54:49 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-15.reflexion.net [208.70.210.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7212EBC5 for ; Fri, 31 Mar 2017 03:54:48 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 17913 invoked from network); 31 Mar 2017 03:54:47 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 31 Mar 2017 03:54:47 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.30.2) with SMTP; Thu, 30 Mar 2017 23:54:46 -0400 (EDT) Received: (qmail 1156 invoked from network); 31 Mar 2017 03:54:46 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 31 Mar 2017 03:54:46 -0000 Received: from [192.168.1.119] (c-67-170-167-181.hsd1.or.comcast.net [67.170.167.181]) by iron2.pdx.net (Postfix) with ESMTPSA id EC716EC7848; Thu, 30 Mar 2017 20:54:45 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Is this procstat -v output valid/expected? Explanation? Message-Id: Date: Thu, 30 Mar 2017 20:54:45 -0700 Cc: freebsd-arm To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Mar 2017 03:54:49 -0000 The following is based on a test-case program that: A) Allocates lots of 14KiByte "regions" with malloc, initializing each byte of each region to a never-zero pattern of bytes. "Lots" uses up most of 256 MiBytes across the regions. Once initialized none of these bytes are written again (not even by the later child process). B) Tests the byte patterns (SIGABRT if the pattern test fails). C) Forks. D) The parent waits for child; child sleeps 60 sec. Note: In the full test forcing swapping of both is involved during the sleep but that is not being done here. E) The child checks the byte patterns and exits (SIGABRT if the pattern test fails). The child does not write to any of the allocation regions. F) The (former) parent checks the byte patterns and exits (SIGABRT if the pattern test fails). It does not write to any of the allocated regions during this activity. [The context happens to be arm64.] Note the two instances of "67306" in the below from what will become the parent process: # procstat -v 6310 PID START END PRT RES PRES REF SHD FLAG = TP PATH 6310 0x10000 0x11000 r-- 1 51 3 1 CN-- = vn /root/c_tests/swaptesting2 6310 0x20000 0x21000 r-x 1 51 3 1 CN-- = vn /root/c_tests/swaptesting2 6310 0x30000 0x40000 rw- 16 0 1 0 C--- = vn /root/c_tests/swaptesting2 6310 0x40000 0x41000 r-- 1 38 2 0 ---- = df=20 6310 0x41000 0x75000 rw- 37 38 2 0 ---- = df=20 6310 0x40030000 0x4004b000 r-x 27 29 59 27 CN-- = vn /libexec/ld-elf.so.1 6310 0x4004b000 0x40052000 rw- 7 7 1 0 ---- = df=20 6310 0x4005a000 0x4005b000 rw- 1 0 1 0 C--- = vn /libexec/ld-elf.so.1 6310 0x4005b000 0x4005c000 rw- 1 1 1 0 ---- = df=20 6310 0x4005c000 0x401b4000 r-x 344 376 59 27 CN-- = vn /lib/libc.so.7 6310 0x401b4000 0x401c3000 --- 0 0 1 0 ---- = df=20 6310 0x401c3000 0x401cf000 rw- 12 0 1 0 C--- = vn /lib/libc.so.7 6310 0x401cf000 0x40202000 rw- 22 67306 2 0 ---- = df=20 6310 0x40400000 0x50e00000 rw- 67284 67306 2 0 ---- = df=20 6310 0xfffffffdf000 0xfffffffff000 rw- 3 3 1 0 ---D = df=20 6310 0xfffffffff000 0x1000000000000 r-x 1 1 35 0 ---- = ph=20 Later after the fork (so child sleeping and parent waiting) it as turned into: # procstat -v 6310 PID START END PRT RES PRES REF SHD FLAG = TP PATH 6310 0x10000 0x11000 r-- 1 51 5 1 CN-- = vn /root/c_tests/swaptesting2 6310 0x20000 0x21000 r-x 1 51 5 1 CN-- = vn /root/c_tests/swaptesting2 6310 0x30000 0x40000 rw- 16 0 1 0 C--- = vn /root/c_tests/swaptesting2 6310 0x40000 0x41000 r-- 1 1 2 0 CN-- = df=20 6310 0x41000 0x75000 rw- 37 37 2 0 CN-- = df=20 6310 0x40030000 0x4004b000 r-x 27 29 60 27 CN-- = vn /libexec/ld-elf.so.1 6310 0x4004b000 0x40052000 rw- 7 0 1 0 C--- = df=20 6310 0x4005a000 0x4005b000 rw- 1 0 2 0 CN-- = vn /libexec/ld-elf.so.1 6310 0x4005b000 0x4005c000 rw- 1 0 1 0 C--- = df=20 6310 0x4005c000 0x401b4000 r-x 344 376 60 27 CN-- = vn /lib/libc.so.7 6310 0x401b4000 0x401c3000 --- 0 0 2 0 CN-- = df=20 6310 0x401c3000 0x401cf000 rw- 12 0 2 0 CN-- = vn /lib/libc.so.7 6310 0x401cf000 0x40202000 rw- 22 22 2 0 CN-- = df=20 6310 0x40400000 0x50e00000 rw- 67284 67284 2 0 CN-- = df=20 6310 0xfffffffdf000 0xfffffffff000 rw- 3 0 1 0 C--D = df=20 6310 0xfffffffff000 0x1000000000000 r-x 1 1 36 0 ---- = ph=20 The child never shows the large PRES figure for the range: 0x401cf000 0x40202000 But for the size of that range the earlier PRES=3D=3D67306 seems odd, as if it spans the following: 0x40400000 0x50e0000 In fact 22+67284=3D=3D67306. Another point that I noticed that the I found SHD stays zero on the memory area spanning the allocations (0x40400000 0x50e00000) and more: (This was during the child's sleep.) # procstat -v 6313 PID START END PRT RES PRES REF SHD FLAG = TP PATH 6313 0x10000 0x11000 r-- 1 51 5 1 CN-- = vn /root/c_tests/swaptesting2 6313 0x20000 0x21000 r-x 1 51 5 1 CN-- = vn /root/c_tests/swaptesting2 6313 0x30000 0x40000 rw- 16 0 1 0 C--- = vn /root/c_tests/swaptesting2 6313 0x40000 0x41000 r-- 1 1 2 0 CN-- = df=20 6313 0x41000 0x75000 rw- 37 37 2 0 CN-- = df=20 6313 0x40030000 0x4004b000 r-x 27 29 60 27 CN-- = vn /libexec/ld-elf.so.1 6313 0x4004b000 0x40052000 rw- 7 0 1 0 C--- = df=20 6313 0x4005a000 0x4005b000 rw- 1 0 2 0 CN-- = vn /libexec/ld-elf.so.1 6313 0x4005b000 0x4005c000 rw- 1 0 1 0 C--- = df=20 6313 0x4005c000 0x401b4000 r-x 344 376 60 27 CN-- = vn /lib/libc.so.7 6313 0x401b4000 0x401c3000 --- 0 0 2 0 CN-- = df=20 6313 0x401c3000 0x401cf000 rw- 12 0 2 0 CN-- = vn /lib/libc.so.7 6313 0x401cf000 0x40202000 rw- 22 22 2 0 CN-- = df=20 6313 0x40400000 0x50e00000 rw- 67284 67284 2 0 CN-- = df=20 6313 0xfffffffdf000 0xfffffffff000 rw- 3 0 1 0 C--D = df=20 6313 0xfffffffff000 0x1000000000000 r-x 1 1 36 0 ---- = ph=20 For: 0x40400000 0x50e00000 (and more) my first thought was that forking would shadow for copy-on-write and so the shadow page count would be non-zero in one or both of the parent vs. child. But Ive never seen procstat -v report such a figure for the range holding the allocations. The REF=3D=3D2 also seems odd: it lasts from before the fork through after it as well, both parent and child processes still existing. It would seem that the REF's are not per-process. Context details: # uname -paKU FreeBSD pine64 12.0-CURRENT FreeBSD 12.0-CURRENT r315914M arm64 = aarch64 1200027 1200027 FYI: the source code is. . . (Ignore comments tied to swapping and its/the "problem" for this question.) # more swap_testing2.c // swap_testing2.c // Built via (c++ was clang++ 4.0 in my case): // // cc -g -std=3Dc11 -Wpedantic -o swaptesting2 swap_testing2.c // -O0 and -O2 also gets the problem. #include // for fork(), sleep(.) #include // for pid_t #include // for wait(.) #include // for raise(.), SIGABRT extern void test_setup(void); // Sets up the memory byte = patterns. extern void test_check(void); // Tests the memory byte patterns. extern void partial_test_check(void); // Tests just [0] of = dyn_regions[0] int main(void) { test_setup(); test_check(); // Before fork() [passes] pid_t pid =3D fork(); int wait_status =3D 0;; // After fork; before waitsleep/swap-out. //if (0=3D=3Dpid) partial_test_check(); // Even the above is sufficient by // itself to prevent failure for // region_size 1u through // 4u*1024u! // But 4u*1024u+1u and above fail // with this access to memory. // The failing test is of // (*dyn_regions[0]).array[4096u]. // This test never fails here. if (0 // for size_t, NULL #include // for malloc(.), free(.) #define region_size (14u*1024u) // Bad dyn_regions patterns, parent and child // processes: // 256u, 2u*1024u, 4u*1024u, 8u*1024u, // 9u*1024u, 12u*1024u, 14u*1024u // (but see the partial_test_check() call // notes above). // Works: // 14u*1024u+1u, 15u*1024u, 16u*1024u, // 32u*1024u, 256u*1024u*1024u #define num_regions (256u*1024u*1024u/region_size) typedef volatile unsigned char value_type; struct region_struct { value_type array[region_size]; }; typedef struct region_struct region; static region * volatile dyn_regions[num_regions] =3D {NULL,}; static value_type value(size_t v) { return (value_type)((v&0xFEu)|0x1u); = } // value now avoids the zero value since the failures // are zeros. void test_setup(void) { for(size_t i=3D0u; i Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CC3ED257EB for ; Sat, 1 Apr 2017 01:05:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id 4A370F40 for ; Sat, 1 Apr 2017 01:05:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from ford.home.vangyzen.net (unknown [76.164.15.242]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 82D105646B; Fri, 31 Mar 2017 20:05:44 -0500 (CDT) Subject: Re: One Priority Per Run Queue To: Warner Losh , "freebsd-hackers@freebsd.org" References: <1aafd6a2-828c-06f5-bdac-e4c953a403b5@FreeBSD.org> From: Eric van Gyzen Message-ID: Date: Fri, 31 Mar 2017 20:05:43 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2017 01:05:45 -0000 On 03/29/2017 16:18, Warner Losh wrote: > How does this scheme prevent starvation of low priority processes? Or > rather, how will this change after this change. I don't know. How does the current scheme do this? I had thought the rationale for assigning four priorities to each run queue was that it was "good enough" and the smaller number of run queues reduced the overhead of the scheduler. Is there a more interesting reason that I'm missing? (This wouldn't be the first time.) Cheers, Eric From owner-freebsd-hackers@freebsd.org Sat Apr 1 04:13:57 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B01CD28D13 for ; Sat, 1 Apr 2017 04:13:57 +0000 (UTC) (envelope-from by@meetlost.com) Received: from meetlost.com (freebsd.meetlost.com [IPv6:2403:2500:8000:1::962]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freebsd103", Issuer "freebsd103" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DCC81160 for ; Sat, 1 Apr 2017 04:13:56 +0000 (UTC) (envelope-from by@meetlost.com) Received: from meetlost.com (localhost [127.0.0.1]) by meetlost.com (8.15.2/8.15.2) with ESMTPS id v310u4hM078723 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 1 Apr 2017 00:56:04 GMT (envelope-from by@meetlost.com) Received: (from by@localhost) by meetlost.com (8.15.2/8.15.2/Submit) id v310u4iB078722 for freebsd-hackers@freebsd.org; Sat, 1 Apr 2017 00:56:04 GMT (envelope-from by) Date: Sat, 1 Apr 2017 00:56:04 GMT From: Yao Bao Message-Id: <201704010056.v310u4iB078722@meetlost.com> To: freebsd-hackers@freebsd.org Subject: make uninstall /usr/src/share/doc/ X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2017 04:13:57 -0000 Hi, I installed the docs under /usr/src/share/doc/ with commands below: make make install make clean And I am thinking about is there anyway to unsintall the docs I just installed? I tried make uninstall but no luck. Can anyone provide some tips on this? Thanks by From owner-freebsd-hackers@freebsd.org Sat Apr 1 12:23:00 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0443FD27C00 for ; Sat, 1 Apr 2017 12:23:00 +0000 (UTC) (envelope-from orka.edison@ovh.fr) Received: from mo29.mail-out.ovh.net (mo29.mail-out.ovh.net [178.32.228.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C51A88E5 for ; Sat, 1 Apr 2017 12:22:59 +0000 (UTC) (envelope-from orka.edison@ovh.fr) Received: from he8.mail.ovh.net (he8.mail.ovh.net [5.135.57.187]) by mo29.mail-out.ovh.net (Postfix) with ESMTP id CAC481D48 for ; Sat, 1 Apr 2017 14:22:57 +0200 (CEST) Received: from DellPrecison (unknown [77.154.202.238]) by he8.mail.ovh.net (Postfix) with ESMTPSA id 7673234FE87 for ; Sat, 1 Apr 2017 14:22:57 +0200 (CEST) Message-ID: <1491049373.5625.1.camel@ovh.fr> Subject: at home server without screen blocked by bad ipfw conf -- live boot usb with sshd From: Orka Edison To: freebsd-hackers@freebsd.org Date: Sat, 01 Apr 2017 14:22:53 +0200 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.3-0ubuntu0.1 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Ovh-Tracer-Id: 11214244548760334554 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrfeelhedrleefgdehtdcutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfqggfjpdevjffgvefmvefgnecuuegrihhlohhuthemuceftddtnecu X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2017 12:23:00 -0000 hi, i brink my at home by a bad ipfw.rules... how can i créate an usb boot key with sshd for access to my server ? with an fixed IP and tools-box for repair my machine. best regard, Orka Edison.