From owner-freebsd-hackers@freebsd.org Thu Sep 19 04:35:24 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D8F5F4B67 for ; Thu, 19 Sep 2019 04:35:24 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail01.asahi-net.or.jp (mail01.asahi-net.or.jp [202.224.55.13]) by mx1.freebsd.org (Postfix) with ESMTP id 46YkY45yFYz4CqK for ; Thu, 19 Sep 2019 04:35:19 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from rv515.advok.com (pool-72-76-119-135.nwrknj.fios.verizon.net [72.76.119.135]) (Authenticated sender: NR2Y-OOT) by mail01.asahi-net.or.jp (Postfix) with ESMTPSA id 13BE885BCD for ; Thu, 19 Sep 2019 13:35:13 +0900 (JST) Date: Thu, 19 Sep 2019 00:34:19 -0400 From: Yoshihiro Ota To: freebsd-hackers@freebsd.org Subject: How to access user process memory/pages from swap_pager.c Message-Id: <20190919003419.007b6de7f9887617f254e334@j.email.ne.jp> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; i386-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46YkY45yFYz4CqK X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ota@j.email.ne.jp designates 202.224.55.13 as permitted sender) smtp.mailfrom=ota@j.email.ne.jp X-Spamd-Result: default: False [0.86 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:202.224.55.0/24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.18)[0.184,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.35)[-0.348,0]; MIME_TRACE(0.00)[0:+]; MV_CASE(0.50)[]; DMARC_NA(0.00)[email.ne.jp]; IP_SCORE(0.83)[ipnet: 202.224.32.0/19(0.21), asn: 4685(3.94), country: JP(-0.02)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[13.55.224.202.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:4685, ipnet:202.224.32.0/19, country:JP]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[135.119.76.72.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2019 04:35:24 -0000 Hi, I'm trying to get some data / statistics for pages being swapped out. I'm trying to access pages being swapped out in swp_pager_putpages. Given I have access to vm_page_t, it looked like I would be able to use VM_PAGE_TO_PHYS() to access user pages. However, I get page-fault panics like: panic: vm_fault_hold: fault on nofault entry, addr: 0 Which functions/setup do I need to access/copy user data being swapped? Thanks, Hiro From owner-freebsd-hackers@freebsd.org Thu Sep 19 06:12:27 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2AE02F7670 for ; Thu, 19 Sep 2019 06:12:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46Ymj61ycmz4JBg for ; Thu, 19 Sep 2019 06:12:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x8J6CHNR072942 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 19 Sep 2019 09:12:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x8J6CHNR072942 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x8J6CGIZ072941; Thu, 19 Sep 2019 09:12:16 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 19 Sep 2019 09:12:16 +0300 From: Konstantin Belousov To: Yoshihiro Ota Cc: freebsd-hackers@freebsd.org Subject: Re: How to access user process memory/pages from swap_pager.c Message-ID: <20190919061216.GG2559@kib.kiev.ua> References: <20190919003419.007b6de7f9887617f254e334@j.email.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190919003419.007b6de7f9887617f254e334@j.email.ne.jp> User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46Ymj61ycmz4JBg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(0.00)[ip: (-2.63), ipnet: 2001:470::/32(-4.47), asn: 6939(-3.23), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2019 06:12:27 -0000 On Thu, Sep 19, 2019 at 12:34:19AM -0400, Yoshihiro Ota wrote: > Hi, > > I'm trying to get some data / statistics for pages being swapped out. > > I'm trying to access pages being swapped out in swp_pager_putpages. > Given I have access to vm_page_t, it looked like I would be able to use VM_PAGE_TO_PHYS() to access user pages. > > However, I get page-fault panics like: > panic: vm_fault_hold: fault on nofault entry, addr: 0 > > Which functions/setup do I need to access/copy user data being swapped? I am not sure what you are trying to do, but whatever is it, it is unreasonable. When a page is swapped out, its physical memory frame, described by vm_page_t, is reused for something else. The only resource consumed by swapped out page is the index in the vm_object page queue, swap space which holds the actual content, and pointer from swap pager data to the swap location. From owner-freebsd-hackers@freebsd.org Thu Sep 19 13:02:53 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DCE36122ED0 for ; Thu, 19 Sep 2019 13:02:53 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46Yxph6jFQz3DM0 for ; Thu, 19 Sep 2019 13:02:52 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from sslproxy05.your-server.de ([78.46.172.2]) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_1) (envelope-from ) id 1iAw58-0004nX-Oo; Thu, 19 Sep 2019 15:02:46 +0200 Received: from [82.100.198.138] (helo=mail.embedded-brains.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1iAw58-000Se1-Jo; Thu, 19 Sep 2019 15:02:46 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 8E0F62A000F; Thu, 19 Sep 2019 15:03:00 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Tm7MVkhp4gFb; Thu, 19 Sep 2019 15:03:00 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id E91C92A1682; Thu, 19 Sep 2019 15:02:59 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7Tmf6qC5QIGB; Thu, 19 Sep 2019 15:02:59 +0200 (CEST) Received: from huber-nb-linux.suse (unknown [192.168.96.161]) by mail.embedded-brains.de (Postfix) with ESMTPSA id CF4552A000F; Thu, 19 Sep 2019 15:02:59 +0200 (CEST) Subject: Re: Problems with port of NVMe support To: Warner Losh Cc: FreeBSD Hackers References: From: Sebastian Huber Message-ID: <9fb441a6-5fab-3950-8b44-558f0f71dba4@embedded-brains.de> Date: Thu, 19 Sep 2019 15:02:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.101.4/25577/Thu Sep 19 10:20:13 2019) X-Rspamd-Queue-Id: 46Yxph6jFQz3DM0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of sebastian.huber@embedded-brains.de designates 85.10.215.148 as permitted sender) smtp.mailfrom=sebastian.huber@embedded-brains.de X-Spamd-Result: default: False [-2.40 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_SEVEN(0.00)[8]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:85.10.215.148]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[embedded-brains.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[148.215.10.85.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; IP_SCORE(-0.10)[ipnet: 85.10.192.0/18(1.29), asn: 24940(-1.79), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:85.10.192.0/18, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; HAS_X_AS(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2019 13:02:53 -0000 On 14/09/2019 07:08, Warner Losh wrote: >=20 > =C2=A0From the NVMe specification I didn't get any hints what the = error > reason could be. Maybe someone has a hint for me. The platform is a > T4240 PowerPC (big-endian). >=20 >=20 > Maybe the physaddr in the SG list for these commands doesn't translate=20 > right? I found the error, it was a misconfiguration of the Read Completion=20 Boundary (RCB). --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Thu Sep 19 21:24:56 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9DC3A12EFE8 for ; Thu, 19 Sep 2019 21:24:56 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail03.asahi-net.or.jp (mail03.asahi-net.or.jp [202.224.55.15]) by mx1.freebsd.org (Postfix) with ESMTP id 46Z8xx3m3Zz4JfT for ; Thu, 19 Sep 2019 21:24:52 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from rv515.advok.com (pool-72-76-119-135.nwrknj.fios.verizon.net [72.76.119.135]) (Authenticated sender: NR2Y-OOT) by mail03.asahi-net.or.jp (Postfix) with ESMTPSA id E0E8668B9D; Fri, 20 Sep 2019 06:24:47 +0900 (JST) Date: Thu, 19 Sep 2019 17:23:53 -0400 From: Yoshihiro Ota To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: How to access user process memory/pages from swap_pager.c Message-Id: <20190919172353.2da06fc2aec3a332e2462ada@j.email.ne.jp> In-Reply-To: <20190919061216.GG2559@kib.kiev.ua> References: <20190919003419.007b6de7f9887617f254e334@j.email.ne.jp> <20190919061216.GG2559@kib.kiev.ua> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; i386-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46Z8xx3m3Zz4JfT X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ota@j.email.ne.jp designates 202.224.55.15 as permitted sender) smtp.mailfrom=ota@j.email.ne.jp X-Spamd-Result: default: False [0.49 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:202.224.55.0/24]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[email.ne.jp]; NEURAL_HAM_LONG(-0.85)[-0.851,0]; NEURAL_SPAM_MEDIUM(0.18)[0.179,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(0.96)[ip: (0.68), ipnet: 202.224.32.0/19(0.21), asn: 4685(3.94), country: JP(-0.02)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[15.55.224.202.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:4685, ipnet:202.224.32.0/19, country:JP]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[135.119.76.72.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2019 21:24:56 -0000 On Thu, 19 Sep 2019 09:12:16 +0300 Konstantin Belousov wrote: > On Thu, Sep 19, 2019 at 12:34:19AM -0400, Yoshihiro Ota wrote: > > Hi, > > > > I'm trying to get some data / statistics for pages being swapped out. > > > > I'm trying to access pages being swapped out in swp_pager_putpages. > > Given I have access to vm_page_t, it looked like I would be able to use VM_PAGE_TO_PHYS() to > > access user pages. > > > > However, I get page-fault panics like: > > panic: vm_fault_hold: fault on nofault entry, addr: 0 > > > > Which functions/setup do I need to access/copy user data being swapped? > > I am not sure what you are trying to do, but whatever is it, it is > unreasonable. > > When a page is swapped out, its physical memory frame, described by > vm_page_t, is reused for something else. The only resource consumed by > swapped out page is the index in the vm_object page queue, swap space > which holds the actual content, and pointer from swap pager data to the > swap location. I'm trying before pageout I/O are scheduled at https://svnweb.freebsd.org/base/head/sys/vm/swap_pager.c?revision=352407&view=markup#l1373 This is where it is about to start looking for available swap spaces in swap devices. Hiro From owner-freebsd-hackers@freebsd.org Fri Sep 20 21:00:36 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CD322127027; Fri, 20 Sep 2019 21:00:36 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46ZmMR62Wyz4ctB; Fri, 20 Sep 2019 21:00:35 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: by mail-io1-xd43.google.com with SMTP id v2so19193688iob.10; Fri, 20 Sep 2019 14:00:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ABHAaOdC0ZGtWKBW0F8L2MVhuSDFYtQmUjNQDZ63w6Y=; b=h+jp59duGEV8CIiml2T88WcViB1lY2MBU6rmKCzkRAJUCWGHd94GzU/tfkarG4Avd5 FbxUth6p3lWCc7I8ZQ/ENXFR5Rv/MqxbZC+UzWOx9zKzm5xr8dNVXBZxUPikf8ln6jaM kdQQQHi9S8/tPSsPCWzdqazRiXTOQ9ts3GQ53wU8/mgIA2PJBQcQZzZpIGtOLEff1USq S+dk+b5nxzuqzpbLaCcP/PdxLU+F7Gd01P81l1sezvF6OHRQM2T/zgttdtSJPDDh7/L3 UDVO1IFZ+M1s7351+ZVvabOcnxykOowC3vk/BedO3IG71fEKCgpXYSsC/2MmGvfm4RWU BZMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ABHAaOdC0ZGtWKBW0F8L2MVhuSDFYtQmUjNQDZ63w6Y=; b=qTHU55iEkB0JFT4J6q9znt5BBULaJvYVTNk1VHREa4+zOVREEW3V1eQ2ytFq6NLMIH 6DdjTkfwUcVfE5Rwci0IoFVnXfiyVdVW8tTNuCoKHYIS1+2APqnzzH1/Zo9vgcuOEDWT riyoRC8s8KQ6hIxk4KGEK5b14C7jia7Az1URjKz/BUMmCC8Hu7dgAlji/jBCMiLRXyw5 xc05eU1weCJRWANCLks+5pz9F++135ja+pDS79k7wtFTJmhnBki/coEtAQHpRSwr4zUb pW+dRDKSFxJy1JcFhLNaf24Z8dEAxb/bn7WSWjCVAma9rdMIFqPdfX3kg1NPdPfT316A uWRg== X-Gm-Message-State: APjAAAWT3jJpoM57rz3mmK8RhzC5svI0TOaLN6qlVUGYxiZhq6eHiz0R lezk7bIrhq037A1RmghgBvM3EqivbnRr6VZ+lPGTwD/5 X-Google-Smtp-Source: APXvYqz/B5GqQGRtwrdZ5vcR68PDQtziGXF4xwiEo8Bh0bkNvFHYavpM/xW5mxCnFCmlls1QG3XCMj/JB343wQymdoU= X-Received: by 2002:a5d:97cf:: with SMTP id k15mr7239670ios.151.1569013234151; Fri, 20 Sep 2019 14:00:34 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a02:9f01:0:0:0:0:0 with HTTP; Fri, 20 Sep 2019 14:00:33 -0700 (PDT) In-Reply-To: References: From: grarpamp Date: Fri, 20 Sep 2019 17:00:33 -0400 Message-ID: Subject: Re: Git/Mtn for FreeBSD, PGP WoT Sigs, Merkel Hash Tree Based To: freebsd-security@freebsd.org Cc: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46ZmMR62Wyz4ctB X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=h+jp59du; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of grarpamp@gmail.com designates 2607:f8b0:4864:20::d43 as permitted sender) smtp.mailfrom=grarpamp@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(0.00)[ip: (2.09), ipnet: 2607:f8b0::/32(-2.65), asn: 15169(-2.20), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.4.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Sep 2019 21:00:36 -0000 For consideration... SVN really may not offer much in the way of native internal self authenticating repo to cryptographic levels of security against bitrot, transit corruption and repo ops, external physical editing, have much signing options, etc. Similar to blockchain and ZFS hash merkle-ization, signing the repo init and later points tags commits, along with full verification toolset, is useful function. https://www.monotone.ca/ https://en.wikipedia.org/wiki/Monotone_(software) https://git-scm.com/ https://en.wikipedia.org/wiki/Git Maintaining the kernel's web of trust https://lwn.net/Articles/798230/ Distributing kernel developer PGP keys via pgpkeys.git https://lkml.org/lkml/2019/8/30/597 Signing patch flow https://lwn.net/Articles/737093/ Compromised security happens https://lwn.net/Articles/464233/ https://security.stackexchange.com/questions/67920/how-safe-are-signed-git-= tags-only-as-safe-as-sha-1-or-somehow-safer https://stackoverflow.com/questions/28792784/why-does-git-use-a-cryptograph= ic-hash-function http://fossil-scm.org/index.html/doc/trunk/www/hashpolicy.wiki https://ericsink.com/vcbe/html/cryptographic_hashes.html https://svn.haxx.se/dev/archive-2015-06/0052.shtml http://git.661346.n2.nabble.com/Verifying-the-whole-repository-td1368311.ht= ml https://shattered.io/ https://www.youtube.com/watch?v=3DG8wQ88d85s4 https://en.wikipedia.org/wiki/Data_degradation https://git-scm.com/docs/git-fsck https://marc.info/?l=3Dgit&m=3D118143549107708 https://en.wikipedia.org/wiki/Comparison_of_version-control_software https://en.wikipedia.org/wiki/Deterministic_compilation https://www.monotone.ca/monotone.html#Trust-Evaluation-Hooks How does one know their entire copy of repo obtained on DVD, "mirror", or elsewhere cryptographically matches the authoritative repo... that any commits were actually signed off on... or that any reproducible builds are even reproducing the main repo... etc... cannot be done without secure crypto infrastructure at the very core. "User also knows that even if someone should break into the shared hosting server and tamper with the database, they won=E2=80=99t be able to inject malicious code into the project, because all revisions are signed by the team members, and he has set his Trust Evaluation Hooks so he doesn=E2=80=99t trust the server key for signing revisions. In monotone, the important trust consideration is on the signed content, rather than on the replication path by which that content arrived in your database." Note also CVS, which some BSD's still use (ahem: Open, Net), is even worse than SVN with zero protection at all in any component regarding this subject. It really time to migrate repo tech to year 2020. From owner-freebsd-hackers@freebsd.org Fri Sep 20 21:04:10 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B490A127726; Fri, 20 Sep 2019 21:04:10 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46ZmRZ0NNgz4dZW; Fri, 20 Sep 2019 21:04:09 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: by mail-io1-xd43.google.com with SMTP id q1so19369670ion.1; Fri, 20 Sep 2019 14:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=LPSI5BzqVlp1Oy96Cdq88l+LJG9kqdV2TFC/6hJtJuE=; b=h8yffhjPaxUN8oBhTeekG7iy/zFiZRF/abiwjA3xDAPexx41JEci9mR7bNWDpOGCk4 6dkD/b48xOVFWyi9dTi39Y4mv2pYOPQVHPep6RuQfjhXVCEI2nSyDyZbzTEWYMmGU7fT hYJs039vUz/ZQdtrApNzOIQa05ZvoiSL/7pSIB1gv43eRTJ9kAII5zDfuqmIZgKbatXI gKuzoAZ1oU3tcrDJ6Sg7uXO+Au66m6LCFHbAwvx2fjSUp4Pciw+mkwhI7XUalR1c2io/ gHcw86ew9Ko6eic3iDqZZSEWUl5BS4Vq+e0Lx8eXmT4SYpKt8Yoc0A/iuVFvbjQLByLA hv2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=LPSI5BzqVlp1Oy96Cdq88l+LJG9kqdV2TFC/6hJtJuE=; b=IRzW5jp0DWpyy4qsfZUBbFN1gkron/WtmRMMp73avjrmYsG6qAHQZuaAOuHanXAufj cFasgAyVkIopO7xw3M1ulTIi63tMnRg8YrBnX2HoY6wJqKyDLKMh1QwIQYq7Zays/dGo zXn0ay7xTrOKhVmtO2I2tv/BjpmE244HF3oV4U1BTdAZjSBjUWkDRl67lPsNm7q9qv9+ jAHhKX0l0F0XmrS86k2aO41OpHChmBeUtz1LO8aWGGW33q+C9DjYB5BrGBXefZrQneIQ 7Tvh0cc51gpRgoslIRGjS49UxSSfUrJOVOiol4VKtxRPefMTQwm9T9fpfsG4UoPq3fL/ +akw== X-Gm-Message-State: APjAAAWinv+7ktdfljiolBelThpA1jNJ9O/LzsDH2xx+z5RWWeMywqLy RlLzhcFqgtCXjx3PHXtxzMgiMKYZtbCA7SNDuGZbeK1T X-Google-Smtp-Source: APXvYqxHlH1D43iOQLGuw1MfAI6zzsagskedwCuAEp0yf6t1t0Pz7VBkQ16wLIxd1q/i6qtIcVE1hGgkwy4S/MZ9s34= X-Received: by 2002:a6b:f80f:: with SMTP id o15mr13032915ioh.174.1569013448487; Fri, 20 Sep 2019 14:04:08 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a02:9f01:0:0:0:0:0 with HTTP; Fri, 20 Sep 2019 14:04:08 -0700 (PDT) In-Reply-To: References: From: grarpamp Date: Fri, 20 Sep 2019 17:04:08 -0400 Message-ID: Subject: Re: Git/Mtn for FreeBSD, PGP WoT Sigs, Merkel Hash Tree Based To: freebsd-security@freebsd.org Cc: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46ZmRZ0NNgz4dZW X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=h8yffhjP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of grarpamp@gmail.com designates 2607:f8b0:4864:20::d43 as permitted sender) smtp.mailfrom=grarpamp@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(0.00)[ip: (2.08), ipnet: 2607:f8b0::/32(-2.65), asn: 15169(-2.20), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.4.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Sep 2019 21:04:10 -0000 [broken links fixed] For consideration... SVN really may not offer much in the way of native internal self authenticating repo to cryptographic levels of security against bitrot, transit corruption and repo ops, external physical editing, have much signing options, etc. Similar to blockchain and ZFS hash merkle-ization, signing the repo init and later points tags commits, along with full verification toolset, is useful function. https://www.monotone.ca/ https://en.wikipedia.org/wiki/Monotone_(software) https://git-scm.com/ https://en.wikipedia.org/wiki/Git Maintaining the kernel's web of trust https://lwn.net/Articles/798230/ Distributing kernel developer PGP keys via pgpkeys.git https://lkml.org/lkml/2019/8/30/597 Signing patch flow https://lwn.net/Articles/737093/ Compromised security happens https://lwn.net/Articles/464233/ https://security.stackexchange.com/questions/67920/how-safe-are-signed-git-= tags-only-as-safe-as-sha-1-or-somehow-safer https://stackoverflow.com/questions/28792784/why-does-git-use-a-cryptograph= ic-hash-function http://fossil-scm.org/index.html/doc/trunk/www/hashpolicy.wiki https://ericsink.com/vcbe/html/cryptographic_hashes.html https://svn.haxx.se/dev/archive-2015-06/0052.shtml http://git.661346.n2.nabble.com/Verifying-the-whole-repository-td1368311.ht= ml https://shattered.io/ https://www.youtube.com/watch?v=3DG8wQ88d85s4 https://en.wikipedia.org/wiki/Data_degradation https://git-scm.com/docs/git-fsck https://marc.info/?l=3Dgit&m=3D118143549107708 https://en.wikipedia.org/wiki/Comparison_of_version-control_software https://en.wikipedia.org/wiki/Deterministic_compilation https://www.monotone.ca/monotone.html#Trust-Evaluation-Hooks How does one know their entire copy of repo obtained on DVD, "mirror", or elsewhere cryptographically matches the authoritative repo... that any commits were actually signed off on... or that any reproducible builds are even reproducing the main repo... etc... cannot be done without secure crypto infrastructure at the very core. "User also knows that even if someone should break into the shared hosting server and tamper with the database, they won=E2=80=99t be able to inject malicious code into the project, because all revisions are signed by the team members, and he has set his Trust Evaluation Hooks so he doesn=E2=80=99t trust the server key for signing revisions. In monotone, the important trust consideration is on the signed content, rather than on the replication path by which that content arrived in your database." Note also CVS, which some BSD's still use (ahem: Open, Net), is even worse than SVN with zero protection at all in any component regarding this subject. It really time to migrate repo tech to year 2020.