From owner-freebsd-hackers@freebsd.org Sun Oct 21 17:10:28 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A57DFFAC6C for ; Sun, 21 Oct 2018 17:10:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id D636E70135 for ; Sun, 21 Oct 2018 17:10:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [94.19.235.70]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id D708768D for ; Sun, 21 Oct 2018 20:10:20 +0300 (MSK) Date: Sun, 21 Oct 2018 20:10:21 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <170994671.20181021201021@serebryakov.spb.ru> To: freebsd-hackers@freebsd.org Subject: What is wrong with dtrace's stack()? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 17:10:28 -0000 Hello Freebsd-hackers, I'm trying to profile strange if_gif and if_gre performance on hardware without pmc. So, I'm sampling kernel stacks with simple dtrace script. And I have a lot of stacks which show something like this: kernel`ipsec_hdrsiz_inpcb+0xa1 kernel`soo_write+0x33 kernel`dofilewrite+0x79 kernel`sys_write+0xc3 kernel`amd64_syscall+0x332 kernel`0xffffffff8086c87d Functions after soo_write could be different, but address is always the same: soo_write+0x33. But soo_write doesn't call all these functions, in first place! soo_write looks like: Dump of assembler code for function soo_write: 0xffffffff8060f930 <+0>: push %rbp 0xffffffff8060f931 <+1>: mov %rsp,%rbp 0xffffffff8060f934 <+4>: push %r15 0xffffffff8060f936 <+6>: push %r14 0xffffffff8060f938 <+8>: push %r12 0xffffffff8060f93a <+10>: push %rbx 0xffffffff8060f93b <+11>: sub $0x10,%rsp 0xffffffff8060f93f <+15>: mov %rsi,%r12 0xffffffff8060f942 <+18>: mov (%rdi),%rbx 0xffffffff8060f945 <+21>: mov 0x28(%r12),%rax 0xffffffff8060f94a <+26>: mov %rax,(%rsp) 0xffffffff8060f94e <+30>: xor %esi,%esi 0xffffffff8060f950 <+32>: xor %ecx,%ecx 0xffffffff8060f952 <+34>: xor %r8d,%r8d .... Now I can not trust all these collected stacks. What do I do wrong?! I have in my kernel config: makeoptions DEBUG=-g makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support options KDTRACE_FRAME # Ensure frames are compiled in options KDTRACE_HOOKS # Kernel DTrace hooks -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-hackers@freebsd.org Sun Oct 21 17:21:56 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F0B4FFB657 for ; Sun, 21 Oct 2018 17:21:56 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CBA1E7092B; Sun, 21 Oct 2018 17:21:55 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io1-xd32.google.com with SMTP id a23-v6so89614iod.7; Sun, 21 Oct 2018 10:21:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FrGLxvY6GXckrJefk0q6QE8KjEILi+Ks0XO3Cq76orU=; b=jDWpfJCXPts9cp/iySiGs+AjJrzCkH2MzLcBi9Mz+6jTrqGwBUSRitl8v80yzUDYSF BDqnSm+KlMQIGglKl/eAOssXCDaw7HGMk5AJTCQn0GRu6yNmgPgOoR++CNNHJPBJi8Gv UGUN8v1ssGjRxAHjCZk7UZ3lqEDGyoXa0Mdu1NFTORnn3BnkNRGvoN4t2RoB49AsJ59h NqEjT3v89FOLPpc7p4VLg82xxD+LbBlgEDuZCLnqBkcp0DUayBc8YWb3E6mZg3PSJ5fv hdNSQSgNPUYeuBgeBlzMHe9EBbV9YWQqw65IAiuxaDvM0ZqvR7pk/JJv9SMREDb+Qco/ VI0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FrGLxvY6GXckrJefk0q6QE8KjEILi+Ks0XO3Cq76orU=; b=TPRpq0TskMcPeXu3ueKGsqeJXafKSdrB/BU9Bb1AkPWxiXx9GwTRgo1xxk6tM242Fv jEL8vCk4tbviCjFww8Dq8+/6U6NsUrdbem+XEXwoCtPlWYd1sIwx1vA7KWzRGZvoiz4S rtesW4jlVTlyKdYxBbUA47NzNFR8Oe2ch+8HFBJKeF1OmGrvpP1brvOORHIcvYsh6HjK zvr1Q7Y20xi4DAP8JFyp0jOy/x+QS4ZdHOrt7LNlPGd0TYeqEvbR8F6Pqw4x8cqWFuRk w2XwOQLr8KmDBbM0Pwdme0nbWwIfMUky67X3HL71dlotNdBHME67AvD7/DzyDWDKbaIl xlTA== X-Gm-Message-State: AGRZ1gKAd2BAhI6NWAenk+4PiDllGF6xboVL3CcS91KUOsgczdfPwwk6 71SNlwrgJwY6Y5U4bxicHqG4/ho74McH/9wA38dj7g== X-Google-Smtp-Source: AJdET5eqL6pKqx07ET5jiIRDY/jeQXiNHEVhVrHTq1eHyaP5wG8x9QOGO1zqOpm8iT8nVV1XftA4r8dZ9o+aca8XqKE= X-Received: by 2002:a6b:e802:: with SMTP id f2-v6mr7034982ioh.19.1540142514886; Sun, 21 Oct 2018 10:21:54 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> In-Reply-To: <170994671.20181021201021@serebryakov.spb.ru> From: Conrad Meyer Date: Sun, 21 Oct 2018 10:21:43 -0700 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: Lev Serebryakov Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 17:21:56 -0000 Your assembler dump offsets are in decimal. Look for offset 0x33 = +51, not +33. Conrad On Sun, Oct 21, 2018 at 10:12 AM Lev Serebryakov wrote: > > Hello Freebsd-hackers, > > I'm trying to profile strange if_gif and if_gre performance on hardware > without pmc. So, I'm sampling kernel stacks with simple dtrace script. > > And I have a lot of stacks which show something like this: > > kernel`ipsec_hdrsiz_inpcb+0xa1 > kernel`soo_write+0x33 > kernel`dofilewrite+0x79 > kernel`sys_write+0xc3 > kernel`amd64_syscall+0x332 > kernel`0xffffffff8086c87d > > Functions after soo_write could be different, but address is always the > same: soo_write+0x33. > > But soo_write doesn't call all these functions, in first place! soo_write > looks like: > > Dump of assembler code for function soo_write: > 0xffffffff8060f930 <+0>: push %rbp > 0xffffffff8060f931 <+1>: mov %rsp,%rbp > 0xffffffff8060f934 <+4>: push %r15 > 0xffffffff8060f936 <+6>: push %r14 > 0xffffffff8060f938 <+8>: push %r12 > 0xffffffff8060f93a <+10>: push %rbx > 0xffffffff8060f93b <+11>: sub $0x10,%rsp > 0xffffffff8060f93f <+15>: mov %rsi,%r12 > 0xffffffff8060f942 <+18>: mov (%rdi),%rbx > 0xffffffff8060f945 <+21>: mov 0x28(%r12),%rax > 0xffffffff8060f94a <+26>: mov %rax,(%rsp) > 0xffffffff8060f94e <+30>: xor %esi,%esi > 0xffffffff8060f950 <+32>: xor %ecx,%ecx > 0xffffffff8060f952 <+34>: xor %r8d,%r8d > .... > > Now I can not trust all these collected stacks. What do I do wrong?! > > I have in my kernel config: > > makeoptions DEBUG=-g > makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support > options KDTRACE_FRAME # Ensure frames are compiled in > options KDTRACE_HOOKS # Kernel DTrace hooks > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Sun Oct 21 17:26:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D6DEFFB998 for ; Sun, 21 Oct 2018 17:26:59 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id E30AA70C59 for ; Sun, 21 Oct 2018 17:26:58 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [94.19.235.70]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 7DDE2696; Sun, 21 Oct 2018 20:26:56 +0300 (MSK) Date: Sun, 21 Oct 2018 20:26:52 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <4210174731.20181021202652@serebryakov.spb.ru> To: Conrad Meyer CC: "freebsd-hackers@freebsd.org" Subject: Re: What is wrong with dtrace's stack()? In-Reply-To: References: <170994671.20181021201021@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 17:26:59 -0000 Hello Conrad, Sunday, October 21, 2018, 8:21:43 PM, you wrote: > Your assembler dump offsets are in decimal. Ooops! > Look for offset 0x33 = +51, not +33. It is call to sosend(), which could call many other things, but why it is missed at stack output!? -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-hackers@freebsd.org Sun Oct 21 21:37:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C219D1037259 for ; Sun, 21 Oct 2018 21:37:34 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5C68379BA4 for ; Sun, 21 Oct 2018 21:37:34 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [94.19.235.70]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 1AD466E8; Mon, 22 Oct 2018 00:37:33 +0300 (MSK) Date: Mon, 22 Oct 2018 00:37:34 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <475670271.20181022003734@serebryakov.spb.ru> To: Conrad Meyer CC: "freebsd-hackers@freebsd.org" Subject: Re: What is wrong with dtrace's stack()? In-Reply-To: References: <170994671.20181021201021@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 21:37:34 -0000 Hello Conrad, Sunday, October 21, 2018, 8:21:43 PM, you wrote: > Your assembler dump offsets are in decimal. Look for offset 0x33 = > +51, not +33. Problem is, sosend() is not very interesting by itself, and looks like several layers of stack are always lost. I see a lot of stacks like this: kernel`lock_delay+0x42 kernel`soo_write+0x33 kernel`dofilewrite+0x79 kernel`sys_write+0xc3 kernel`amd64_syscall+0x332 kernel`0xffffffff8086c87d But event sosend() doesn't call lock_delay(), so it is impossible to understand why do lock_delay() seen 41932 times in 60 seconds at top of the stack. Where are all call stack?! All these functions could not be inlined, as sosend() is located in other translation unit and it calls function by pointer, this call could not be inlined too. -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-hackers@freebsd.org Sun Oct 21 21:46:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0CB310375D1 for ; Sun, 21 Oct 2018 21:46:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 538B579FF1; Sun, 21 Oct 2018 21:46:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f177.google.com with SMTP id x3-v6so35132463lji.13; Sun, 21 Oct 2018 14:46:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3ekOnP5FYMvK2rIgqcb/GI1Cg0GIl2Cqix8tbhwbEm8=; b=lTIBAygcX+QdvgxfL9f9wjpD2GI+vELWiZ4o5VwtqeUqHHDvbbXAtUfuNMBFqZX8O9 DUrSviTY825QGU2XJuV4c1pZp0ERuOK1XlRdvG5Gf4a968PRuZ0cDnvtzM6PJR3X5ooa tGV4StKvAS2w0fI6EI9M5TiFRjMjiMUgoaj2Eay4+x/DsJRdHS8qrBKXX2uxcayaX/nB BR42JB1FSKgy/cMnFkv5Yr8ZrL0z40w6nYZS/dcLhlBBhea4OnzABsqx2wS1S4CMv2c3 vj/eyFfB6VQDYSCE7oWXHiiWGT8jbsVSFmiWLh6eKab1VYZLTDqwPzgfvbnjcF+b5XEt cORQ== X-Gm-Message-State: ABuFfogFTr99Q/C30R93XtArT9ZI0/HAca0/sMPQm1p/P8yZEstNZXVC AkWQE2FGIHDjxb+3j5eDoFhezwy7LgFxiyGr+pKjotsS X-Google-Smtp-Source: ACcGV62es0S3kliv0OJ/PJ9SYSlKKGhZauGXaVPsO/LAsuqfkGuEHpenQ3FyAnIUzhSW5RXLi7/Ei8vu/YWfZyc9plc= X-Received: by 2002:a2e:9c89:: with SMTP id x9-v6mr28058342lji.110.1540156957602; Sun, 21 Oct 2018 14:22:37 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <4210174731.20181021202652@serebryakov.spb.ru> In-Reply-To: <4210174731.20181021202652@serebryakov.spb.ru> From: Alan Somers Date: Sun, 21 Oct 2018 15:22:25 -0600 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: Lev Serebryakov Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 21:46:09 -0000 On Sun, Oct 21, 2018 at 11:27 AM Lev Serebryakov wrote: > Hello Conrad, > > Sunday, October 21, 2018, 8:21:43 PM, you wrote: > > > Your assembler dump offsets are in decimal. > Ooops! > Also, rather than inspecting the assembly, you can fire up kgdb and type (kgdb) list *(soo_write+0x33) And it will tell you the corresponding source line. > > > Look for offset 0x33 = +51, not +33. > It is call to sosend(), which could call many other things, but why it is > missed at stack output!? > dtrace doesn't have visibility into functions that get automatically inlined. So effectively it leaves out some stack frames. Also, dtrace doesn't indicate when a function got called via a function pointer as opposed to a direct call. ipsec_hdrsiz_input, for example, gets called only via function pointers, referenced by the IPSEC_HDRSIZE macro. That, in turn, is called by tcp_output. Finally, if the compiler uses a tail call optimization, then dtrace won't see the tail caller's stack frame. In your case, it looks like there are several layers that are invisible to dtrace. The true stack probably looks something like this: soo_write->sosend->sosend_generic (via pru_sosend function pointer)->tcp_usr_send(via pru_send function pointer)->tcp_output(via tfb_tcp_output function pointer)->IPSEC_HDRSIZE->ipsec_hdrsiz_inpcb (via hdrsiz function pointer) Hope that helps. -Alan > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sun Oct 21 21:47:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6C6D1037654 for ; Sun, 21 Oct 2018 21:47:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CADF7A0BC; Sun, 21 Oct 2018 21:47:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f41.google.com with SMTP id l1-v6so9064974lfc.3; Sun, 21 Oct 2018 14:47:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cQ/MaefgTJ8AP9zpAUvm1pangjFtCx2xExMZZ5KRi7M=; b=RGAcpWz4olp1HuxLF9ri0LbaWlXXD4KhKoZcb/MV/JOAkvbujfQQ9t1+IC8OmNc+ec jBAQb2S3sLKC5fwXFNFxrDxbfHCU6GgvIQOz+UBvRRQFgZlBtlC3nsKbYHMZ+161V4p2 Wfga718TRxJtUgLN9H3vtAPXiEJDONSIJ1JEv2XECgWIgQgTZCKo2js7GH0J8bbHa/Y+ x0qU6dw9vlxCogthAg/67wQYGB9aoQK2JVLgajqCGdjnxpJ7i2jQzJVkJhhpJ41ZK6z5 nSbVDtRiMafMQIA1SnaUISoZyxB+lIe7HWT4E7tA65XbS6V0ulQDmzPawML6IDRu7Hl5 N9rw== X-Gm-Message-State: ABuFfoifmaT1HCcKMmp2B1RZCtIrk+4iWnRXuk1VflBGient9prkpEVj CP0SpsZujrifMtfJHBYzqEertfzCrUtp+Ww46NciBw== X-Google-Smtp-Source: ACcGV61hc+gIIzzYMw7Po2n0w8BqxMGaosQIfmz6V33yiHGZl1UUmu0C0eR2j6UsU6dXA9bCVJwTaLQQv+SnKS7A8VU= X-Received: by 2002:a19:964e:: with SMTP id y75-v6mr7871756lfd.58.1540158442187; Sun, 21 Oct 2018 14:47:22 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> In-Reply-To: <475670271.20181022003734@serebryakov.spb.ru> From: Alan Somers Date: Sun, 21 Oct 2018 15:47:10 -0600 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: Lev Serebryakov Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 21:47:24 -0000 On Sun, Oct 21, 2018 at 3:38 PM Lev Serebryakov wrote: > Hello Conrad, > > Sunday, October 21, 2018, 8:21:43 PM, you wrote: > > > Your assembler dump offsets are in decimal. Look for offset 0x33 = > > +51, not +33. > Problem is, sosend() is not very interesting by itself, and looks like > several layers of stack are always lost. > > I see a lot of stacks like this: > > kernel`lock_delay+0x42 > kernel`soo_write+0x33 > kernel`dofilewrite+0x79 > kernel`sys_write+0xc3 > kernel`amd64_syscall+0x332 > kernel`0xffffffff8086c87d > > But event sosend() doesn't call lock_delay(), so it is impossible to > understand why do lock_delay() seen 41932 times in 60 seconds at top of the > stack. Where are all call stack?! All these functions could not be inlined, > as sosend() is located in other translation unit and it calls function by > pointer, this call could not be inlined too. > If you're sure that the function isn't inlined, then it might be using the tail-call optimization instead. That would also explain the missing stack frames, too. If you can manually narrow the options down to a few possible callers, then you could try adding a few SDT probes. That's what I usually do in cases like this. Or, for static functions that are inlined, you can remove the static keyword to (usually) prevent the inlining. -Alan > > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Mon Oct 22 11:45:28 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A253FF80E0 for ; Mon, 22 Oct 2018 11:45:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id A964175FFE; Mon, 22 Oct 2018 11:45:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 790EA79E; Mon, 22 Oct 2018 14:45:26 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Alan Somers Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Date: Mon, 22 Oct 2018 14:45:19 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 11:45:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd Content-Type: multipart/mixed; boundary="22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Alan Somers Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> In-Reply-To: --22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 22.10.2018 0:47, Alan Somers wrote: > =C2=A0I see a lot of stacks like this: >=20 > =C2=A0kernel`lock_delay+0x42 > =C2=A0kernel`soo_write+0x33 > =C2=A0kernel`dofilewrite+0x79 > =C2=A0kernel`sys_write+0xc3 > =C2=A0kernel`amd64_syscall+0x332 > =C2=A0kernel`0xffffffff8086c87d >=20 > =C2=A0But event sosend() doesn't call lock_delay(), so it is imposs= ible to > understand why do lock_delay() seen 41932 times in 60 seconds at to= p > of the > stack. Where are all call stack?! All these functions could not be > inlined, > as sosend() is located in other translation unit and it calls > function by > pointer, this call could not be inlined too. >=20 >=20 > If you're sure that the function isn't inlined, then it might be using > the tail-call optimization instead.=C2=A0 That would also explain the m= issing > stack frames, too. I know about inlining at TCO, but it is not the case for sure. Problem is, they can not be inlined it TCO'ed. soo_wrtite() calls sosend(): error =3D sosend(so, 0, uio, 0, 0, 0, uio->uio_td); if (error =3D=3D EPIPE && (so->so_options & SO_NOSIGPIPE) =3D=3D 0) { PROC_LOCK(uio->uio_td->td_proc); tdsignal(uio->uio_td, SIGPIPE); PROC_UNLOCK(uio->uio_td->td_proc); } It could not be TCO'ed and sosend() is in another translation unit, so it could not be inlined. sosend() calls protocol-specific handler via function pointer, so it should be true call: CURVNET_SET(so->so_vnet); if (!SOLISTENING(so)) error =3D so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, top, control, flags, td); else { m_freem(top); m_freem(control); error =3D ENOTCONN; } CURVNET_RESTORE(); return (error); These frames MUST be here... > If you can manually narrow the options down to a few > possible callers, then you could try adding a few SDT probes.=C2=A0 Tha= t's > what I usually do in cases like this.=C2=A0 Or, for static functions th= at are > inlined, you can remove the static keyword to (usually) prevent the > inlining. Narrowing all paths, which leads to very generic lock_delay()... Uh-oh.= --=20 // Lev Serebryakov --22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T-- --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvNuFBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+33A/+P+bUbYn+N6DMGvCNsJN89TmU4V/TbEZVwIFMDRw+Lkoygf0m2/Q9ts1O 8eFmOGtCBEryiUIfp8DG9XsOxCal/AH+4JNy5C52cXY/lwfZRVYwltBfBm+0DzQC +coHrc6IA9aRvMQc04esZ6wf/GMY47mN1fYc5kr6jou74BHe//VVnWcIPQ1NumHK 5MEGUr+dD7eNWH6gMv0YkwU/mIK7ebz1rscdjUlAjba5l1r8PU8lR6MWl7cuUb1t 7Rec5waJ4E9LoayPojyTIQvKx0rELpML+w0wlCCnqpuEn8NcpvAe9cCkCipc68K4 Vx0fGN+1MQRRab0FkTXvpR+TeiWskDaLktO/o2QIY5rac+6xeqVHzSncY27ydkNj x7F39Swh2Yr217h6gS68iMPjf+N2Jn/+IolLiCacXSY3lwd7qnc0YZnWV6Rxnwqo CnSgRlV5OMt1M8iCKiko0G3fpEdPOwlt1BTha0mL6zh4ybRvf8AjZvBHqDKmvClj bwzARN0ZaAqu2EtRf8Cf43zRdrNeHTwesYn13l66n60AuoWA48aHclgu7Vqp/Gzs eF51WUuiLB4g865s4KRyyy++g0fS0WQGnx8YeP+df7zJQ0TUC2pYRUqu9OTdQy2t /yqtMqtMjbKbT3wU0uJX19EWCQS9FY5nvztoKUt5S5chl/PZRuQ= =/2k9 -----END PGP SIGNATURE----- --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd-- From owner-freebsd-hackers@freebsd.org Mon Oct 22 11:57:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0130FF88DD for ; Mon, 22 Oct 2018 11:57:51 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A24C766E0; Mon, 22 Oct 2018 11:57:51 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 43DDE8D4A163; Mon, 22 Oct 2018 11:57:49 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 686FBD1F8F7; Mon, 22 Oct 2018 11:57:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id xseTz566_p7I; Mon, 22 Oct 2018 11:57:46 +0000 (UTC) Received: from [192.168.1.88] (fresh-ayiya.sbone.de [IPv6:fde9:577b:c1a9:f001::2]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 4A310D1F8F3; Mon, 22 Oct 2018 11:57:46 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Mark Johnston" Cc: freebsd-hackers@freebsd.org Subject: Re: [CFT] capsicum patches for rtsol(8) and rtsold(8) Date: Mon, 22 Oct 2018 11:57:44 +0000 X-Mailer: MailMate (2.0BETAr6123) Message-ID: <2A564C8A-FB64-4D2A-9E3E-392F1FCA66BD@lists.zabbadoz.net> In-Reply-To: <20181016200414.GD5066@raichu> References: <20181015194212.GA2751@spy> <20181016165308.GB5066@raichu> <86D87437-BD34-489A-87B7-33F1089080EE@lists.zabbadoz.net> <20181016200414.GD5066@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 11:57:51 -0000 On 16 Oct 2018, at 20:04, Mark Johnston wrote: > On Tue, Oct 16, 2018 at 06:29:49PM +0000, Bjoern A. Zeeb wrote: >> On 16 Oct 2018, at 16:53, Mark Johnston wrote: >> >>> On Tue, Oct 16, 2018 at 04:06:43PM +0000, Bjoern A. Zeeb wrote: >>>> On 15 Oct 2018, at 19:42, Mark Johnston wrote: >>>> >>>>> https://people.freebsd.org/~markj/patches/rtsold_capsicum.diff >>>> >>>> (0) the git rename doesn’t really work when applying the diff >>>> with >>>> FreeBSD’s patch so the mv has to be done manually >>>> >>>> (1) the rtsol Makefile also needs cap_syslog and util to link to >>>> otherwise rtsold.c has unresolved symbols >>>> >>>> (2) rtsol seem to have worked when manually invoked; >>>> /etc/resolv.conf >>>> was created (I had rm’ed it) and the 3 nameserver lines >>>> re-appeared; >>>> sorry can’t test the search string here >>>> >>>> (3) rtsold crashes: >>> >>> Thanks. I made some last-minute changes and forgot to retest, of >>> course. :( >>> >>> I uploaded a new patch which should fix all of these issues - could >>> you >>> give it a try? >> >> With the old and new patch: >> >> root@i386-a3-carp:/usr/src/sbin/rtsol # rtsol vtnet0 >> failed to run script: Invalid argument >> >> Hadn’t noticed that before. > > That's a cosmetic bug. I uploaded a new patch which should fix it. Same URL? I’d try to test that tomorrow then. >> Also on a running system: >> >> root@i386-a3-carp:/ # rm /etc/resolv.conf >> root@i386-a3-carp:/ # cat /etc/resolv.conf >> cat: /etc/resolv.conf: No such file or directory >> root@i386-a3-carp:/ # sh /etc/rc.d/rtsold restart >> Stopping rtsold. >> Waiting for PIDS: 1047. >> Starting rtsold. >> root@i386-a3-carp:/ # cat /etc/resolv.conf >> cat: /etc/resolv.conf: No such file or directory > > resolvconf -a will only update /etc/resolv.conf if the info in > /var/run/resolvconf/interfaces/vtnet0 has changed, I believe. Try > deleting that file too, and then try running rtsol. When I deleted /etc/resolv.conf and then rtsol manually it had re-appeared. Unclear to me what was in /var/run; I just wanted to point out the difference in behaviour; maybe you are right; I’ll go and check if deleting in /var/run/ as well makes a difference. From owner-freebsd-hackers@freebsd.org Mon Oct 22 12:09:12 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36C60FF9C76 for ; Mon, 22 Oct 2018 12:09:12 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B41307727F for ; Mon, 22 Oct 2018 12:09:10 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w9MC927r008761 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 22 Oct 2018 14:09:03 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w9MC91q6073392 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Mon, 22 Oct 2018 19:09:01 +0700 (+07) (envelope-from eugen@grosbein.net) To: Freebsd hackers list From: Eugene Grosbein Subject: shell read built-in Message-ID: <2fff355e-a729-d5d3-8199-f2bbba80c112@grosbein.net> Date: Mon, 22 Oct 2018 19:08:58 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 12:09:12 -0000 Hi! $ echo test | read line; echo $? 0 $ echo -n test | read line; echo $? 1 Is it right behavior for /bin/sh? This makes it hard for "while read line ... < file" to process last line if it is not followed by new line character. Eugene From owner-freebsd-hackers@freebsd.org Mon Oct 22 12:30:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22101FFA730 for ; Mon, 22 Oct 2018 12:30:51 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B674477CC5 for ; Mon, 22 Oct 2018 12:30:50 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id E357F2204E; Mon, 22 Oct 2018 08:30:43 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 22 Oct 2018 08:30:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm3; bh=j INON/ybj6Rafll1/2sas9rL9HbbstWOxukJRlgy+7Q=; b=UiOCg495yR7V/N2oa NbVgtmbO1MHAradulHPg0Ts0eiH63fUcirLOIwo3ivjETD0Yox7j1WZq6ga2Auvl PWjRnyyMHLWpLKU+J6lMd/eyKUc8HfiHKvgQlx6am9PyLTLoEggT6V2Jx33xN4Ye ozrlDUS/pofnWOC1H9pWcZ2o04MugcWK+OC3G5DtYpKcC/2FRxtQMKteC+go5iwZ 49J35CLEs0OE/GzRTAm80PjVFut4gOWvQkSiY4D0JHOAUBQNf+cSsX6V3zowlspo 0P6JI9JXpVGGNptG9/xx+drvzI0PkerhM2Qmtzh6NnJpA1JGOlSyoJl213WvT/sk lQ23w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=jINON/ybj6Rafll1/2sas9rL9HbbstWOxukJRlgy+ 7Q=; b=QSfle23hYd+uU9rotx4tPgqOJg6eLSaZkZLV8Zs4OXH1SGSlO+gP7z8Fe o5AHgjqvaZGk8jR5VroLXbaiFXuiptWeZFJwEvZlcJpfGtqvoNQ42P9adQiB81KW OxZ8G2kOo+u0KjyNsfr3RiW822ESmJOYyhmqfRGvKExibl1bv5SJhDNNQAWBItgz hwe15BhTbEw+NJoEr3BB14ZuIFCx28CV/jJlHEH6TYLhScBrv/KHGV8kfl6kWptm CjRoJJrItOH0gdDwysV2fcKcaV6J3JDmSZC4N3E/o8vPcjdg+9pX76Uw3xQRs3GN ETsd5lxqqZe+0HFw6l+TxNTf910MQ== X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [5.138.166.47]) by mail.messagingengine.com (Postfix) with ESMTPA id 0D33A102E4; Mon, 22 Oct 2018 08:30:41 -0400 (EDT) Subject: Re: shell read built-in To: Eugene Grosbein , Freebsd hackers list References: <2fff355e-a729-d5d3-8199-f2bbba80c112@grosbein.net> From: Yuri Pankov Openpgp: preference=signencrypt Autocrypt: addr=yuripv@yuripv.net; keydata= xsBNBFu8u6IBCADB11gP0QwnorrHjqAtKLHKHNHskhy0s7jqJKfx0YqXgVBKGLJ9/mjLAz0F CBNvemHSDDTs0mEZ9cBKKi6cmsav6+UQgr//yai6hvXLBJqKchSFO4MhmdvBtsGFq1yKz5Zi uhjmimKyIpgBgvMdbgGbGq6cnSB2uEPmZuJr419SVRODOkXukU+F5WHgaHzDdHAIu1asCt2B +6msxqIqlFWcXyZyTGicTGGvC/PFIsVRUtD1dIJANTC876g7DTb7LZXWiWwJpSJ4GKMXMHVX Ct9BoQ4i3nhKbOxb6Io1wsy+NFyWsTJ9KYrxKKPJP3oG8BWb/cqlFqnE4eNSsiq2q7krABEB AAHNH1l1cmkgUGFua292IDx5dXJpcHZAeXVyaXB2Lm5ldD7CwJcEEwEIAEECGwMFCQWjmoAF CwkIBwMFFQoJCAsFFgMCAQACHgECF4AWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW708wAIZ AQAKCRCTi+B/sSrhbPxBB/961alcU091O+yKT5/oReHVc/PX0Tz4sW3V44AcgLfYlrZavCro EFz90qmCrl0xqEwuAKcC4bjmL8SjPWAhSN6IH9nxdw+HeZnAPiHm/q679Bu47+nHBl3qD/9p +t1PkKeKZfaWToFMt1nq06ytSu6VLMCwLdlDNe6DReX0ex/afEqKsuaIZSKL4UYjRwklp8PU Uf98QkrfapyHB67hQMzfI4tPeJaYyv0cTgfq3kUWJx1V6Xi0b6Zxj4ZrB2TXvaMO5g7yhU9E E3WWAvoe4FgB3a7dHe8atnHhq5+Cuvm6+LD4Jh7jvMAE5UMN+xxQpnGpNghHjaCy4vXrLRBZ nhRYzsBNBFu8u6IBCADKih3Q933rDNj4ZA8FhBQ2RlmBgvwOLcDPIL3h0V7h38y3+HisgFSc XACDsdrTlYZ1bRXkD9FHENynBcv0l/3uGJDk8jaGIDE0TP8OQBRp+IaU9/BHnAqrKxTJGIol Dahy2m+yx2yhdc6B4ujWMDqCF1rWOD+ymOWw+VLllOkrHcZa5PJtX9UOGbApZl8ZTM8El4CA NN8F1bg9MWzUi+8LYoGWGc+BwsFS1OUB1c4SPgMu5fD4Wfsr9yRl06fdpEA2YT7B/j5/5RSC 0sE2Zs/tmJ/JRflHJ12ycj59ma2xQMfEJF40hZDpMFQmZvbVqgEg3ocQcltjbxlIKZ/mjC4z ABEBAAHCwHwEGAEKACYWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW7y7ogIbDAUJBaOagAAK CRCTi+B/sSrhbIDcCACqAZMcoxUBLZa40a5b24j5i1jplvCYYb3h+Q5lt5+BFJ87kCb4dJuU D3kh2i29BrxWQWa9WNue9ozxeYkbkfXubQYXexVolRsnh64OdGsE8KvorBFBB3zdK/GRt2Jy +jsnTfUWuQllbzMP0MfhCDMk1Mo8WvDH2/cOEP/yLKf20a+cd6nLs7bidjmGXo9pyuBKAtV6 Kv+VRu54AL+A/UBYu/eB3Dtvzcnut+1Zq6KaP++kUwPwINLIk04OBDwN0zRNTiqMAFYYyz2v ZHBB6E1th/l//ZC5b9Dk0ZpFI1bYdL9ymnrZe1MqbGPnDCToQxu00T/pZCm6Z92YrZQYuNwl Message-ID: <4ad1ea37-4714-a104-89a2-d7aef70d0f89@yuripv.net> Date: Mon, 22 Oct 2018 15:30:40 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <2fff355e-a729-d5d3-8199-f2bbba80c112@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 12:30:51 -0000 Eugene Grosbein wrote: > Hi! > > $ echo test | read line; echo $? > 0 > $ echo -n test | read line; echo $? > 1 > > Is it right behavior for /bin/sh? > This makes it hard for "while read line ... < file" > to process last line if it is not followed by new line character. POSIX defines the exit status for read as (and sh(1) says the same): The following exit values shall be returned: 0 Successful completion. >0 End-of-file was detected or an error occurred. ...and looks like all of /bin/sh, bash, ksh93 seem to agree on the behavior here. From owner-freebsd-hackers@freebsd.org Mon Oct 22 13:30:07 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D73EFFBE5E for ; Mon, 22 Oct 2018 13:30:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 174F079D44; Mon, 22 Oct 2018 13:30:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1gEaHL-00054P-Ri; Mon, 22 Oct 2018 16:29:55 +0300 Date: Mon, 22 Oct 2018 16:29:55 +0300 From: Slawa Olhovchenkov To: Lev Serebryakov Cc: Alan Somers , "freebsd-hackers@freebsd.org" , cse.cem@gmail.com Subject: Re: What is wrong with dtrace's stack()? Message-ID: <20181022132955.GB1809@zxy.spb.ru> References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 13:30:07 -0000 On Mon, Oct 22, 2018 at 02:45:19PM +0300, Lev Serebryakov wrote: > sosend() calls protocol-specific handler via function pointer, so it > should be true call: > > CURVNET_SET(so->so_vnet); > if (!SOLISTENING(so)) > error = so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, > top, control, flags, td); > else { > m_freem(top); > m_freem(control); > error = ENOTCONN; > } > CURVNET_RESTORE(); > return (error); > > These frames MUST be here... (kgdb) x/80i sosend 0xffffffff8054d6b0 : push %rbp 0xffffffff8054d6b1 : mov %rsp,%rbp 0xffffffff8054d6b4 : mov 0x20(%rdi),%rax 0xffffffff8054d6b8 : mov 0x58(%rax),%rax 0xffffffff8054d6bc : mov 0x98(%rax),%rax 0xffffffff8054d6c3 : pop %rbp 0xffffffff8054d6c4 : jmpq *%rax From owner-freebsd-hackers@freebsd.org Mon Oct 22 13:32:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3314EFFC104 for ; Mon, 22 Oct 2018 13:32:08 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C70CD7A0BD for ; Mon, 22 Oct 2018 13:32:07 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id E3EC221F37; Mon, 22 Oct 2018 09:32:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 22 Oct 2018 09:32:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= subject:from:to:references:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm3; bh=u RykhZCap1Uzgc1T2YlrteIQ0lQCSJQvL6bPiGXB3sI=; b=S5xLajj3AszR70E5G CWaFQO4XvbSazW1RE19OZ7r0MQnvBdfQ8AkOpjVFeb4uj5xRwmPbILJ8roapaQ3i tNgMvQO96x/Dfz31xW39+OblgTHbxF0PFysLJTxZxXYl9a9t/deNzLOsLm36DRea oERwLiBQDLFONFFP/0533345podkZatC4yrPldjigkKL+6G4vFsqk/ARehctrBTj MoHQP1c5K3VyRNLVxoznRAunSUsMBKG942fa+Tr7ziBm3Kdufh55OG0jXlM9Xul/ a3fvLLQeev4rG3ZhMBqhIZqSKJcmY2bS05Vk03t1DmsjgmCxqtUkdVXG9zL2VX0h vncoQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=uRykhZCap1Uzgc1T2YlrteIQ0lQCSJQvL6bPiGXB3 sI=; b=WcdkAhkcrqZzkOD/kv/Utid/F+AI73UtugShwQkzpU9dmc03fV1/+dur3 KxyV1ZMUyxzrl/aWPWd+1iKiPsxR3Pw/ak7cHChaF1yGmsPOFBw2PqL1dwPksTE2 u7FaVcWadjK5vWDiDQ0iQSyJDPCIYc0zhMVPNL9xPxZeuHTFQZlAh+lMt/QBABgo gIzEWSLQIxhS0hB9MkAbr6HUkMKR3diRGdyO9wNwCdqJBhXXfAdkCfbO7XhN4eVp uf2oFkwlvsEuJ+K2y4J2E6HeunhZkQfkw/vpyH9658QFP8JBPwCIPiC7AGIc/zUa yQdrvV2LPcfDeuAkAp6e2Ch75aPmQ== X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [5.138.166.47]) by mail.messagingengine.com (Postfix) with ESMTPA id 646C0102E4; Mon, 22 Oct 2018 09:32:04 -0400 (EDT) Subject: Re: shell read built-in From: Yuri Pankov To: Eugene Grosbein , Freebsd hackers list References: <2fff355e-a729-d5d3-8199-f2bbba80c112@grosbein.net> <4ad1ea37-4714-a104-89a2-d7aef70d0f89@yuripv.net> Openpgp: preference=signencrypt Autocrypt: addr=yuripv@yuripv.net; keydata= xsBNBFu8u6IBCADB11gP0QwnorrHjqAtKLHKHNHskhy0s7jqJKfx0YqXgVBKGLJ9/mjLAz0F CBNvemHSDDTs0mEZ9cBKKi6cmsav6+UQgr//yai6hvXLBJqKchSFO4MhmdvBtsGFq1yKz5Zi uhjmimKyIpgBgvMdbgGbGq6cnSB2uEPmZuJr419SVRODOkXukU+F5WHgaHzDdHAIu1asCt2B +6msxqIqlFWcXyZyTGicTGGvC/PFIsVRUtD1dIJANTC876g7DTb7LZXWiWwJpSJ4GKMXMHVX Ct9BoQ4i3nhKbOxb6Io1wsy+NFyWsTJ9KYrxKKPJP3oG8BWb/cqlFqnE4eNSsiq2q7krABEB AAHNH1l1cmkgUGFua292IDx5dXJpcHZAeXVyaXB2Lm5ldD7CwJcEEwEIAEECGwMFCQWjmoAF CwkIBwMFFQoJCAsFFgMCAQACHgECF4AWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW708wAIZ AQAKCRCTi+B/sSrhbPxBB/961alcU091O+yKT5/oReHVc/PX0Tz4sW3V44AcgLfYlrZavCro EFz90qmCrl0xqEwuAKcC4bjmL8SjPWAhSN6IH9nxdw+HeZnAPiHm/q679Bu47+nHBl3qD/9p +t1PkKeKZfaWToFMt1nq06ytSu6VLMCwLdlDNe6DReX0ex/afEqKsuaIZSKL4UYjRwklp8PU Uf98QkrfapyHB67hQMzfI4tPeJaYyv0cTgfq3kUWJx1V6Xi0b6Zxj4ZrB2TXvaMO5g7yhU9E E3WWAvoe4FgB3a7dHe8atnHhq5+Cuvm6+LD4Jh7jvMAE5UMN+xxQpnGpNghHjaCy4vXrLRBZ nhRYzsBNBFu8u6IBCADKih3Q933rDNj4ZA8FhBQ2RlmBgvwOLcDPIL3h0V7h38y3+HisgFSc XACDsdrTlYZ1bRXkD9FHENynBcv0l/3uGJDk8jaGIDE0TP8OQBRp+IaU9/BHnAqrKxTJGIol Dahy2m+yx2yhdc6B4ujWMDqCF1rWOD+ymOWw+VLllOkrHcZa5PJtX9UOGbApZl8ZTM8El4CA NN8F1bg9MWzUi+8LYoGWGc+BwsFS1OUB1c4SPgMu5fD4Wfsr9yRl06fdpEA2YT7B/j5/5RSC 0sE2Zs/tmJ/JRflHJ12ycj59ma2xQMfEJF40hZDpMFQmZvbVqgEg3ocQcltjbxlIKZ/mjC4z ABEBAAHCwHwEGAEKACYWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW7y7ogIbDAUJBaOagAAK CRCTi+B/sSrhbIDcCACqAZMcoxUBLZa40a5b24j5i1jplvCYYb3h+Q5lt5+BFJ87kCb4dJuU D3kh2i29BrxWQWa9WNue9ozxeYkbkfXubQYXexVolRsnh64OdGsE8KvorBFBB3zdK/GRt2Jy +jsnTfUWuQllbzMP0MfhCDMk1Mo8WvDH2/cOEP/yLKf20a+cd6nLs7bidjmGXo9pyuBKAtV6 Kv+VRu54AL+A/UBYu/eB3Dtvzcnut+1Zq6KaP++kUwPwINLIk04OBDwN0zRNTiqMAFYYyz2v ZHBB6E1th/l//ZC5b9Dk0ZpFI1bYdL9ymnrZe1MqbGPnDCToQxu00T/pZCm6Z92YrZQYuNwl Message-ID: <4cd38621-7387-6302-437e-b8fda3fbe1cd@yuripv.net> Date: Mon, 22 Oct 2018 16:32:02 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <4ad1ea37-4714-a104-89a2-d7aef70d0f89@yuripv.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 13:32:08 -0000 Yuri Pankov wrote: > Eugene Grosbein wrote: >> Hi! >> >> $ echo test | read line; echo $? >> 0 >> $ echo -n test | read line; echo $? >> 1 >> >> Is it right behavior for /bin/sh? >> This makes it hard for "while read line ... < file" >> to process last line if it is not followed by new line character. > > POSIX defines the exit status for read as (and sh(1) says the same): > > The following exit values shall be returned: > 0 Successful completion. > >0 End-of-file was detected or an error occurred. > > ...and looks like all of /bin/sh, bash, ksh93 seem to agree on the > behavior here. BTW, it looks like last line is still parsed despite not having \n, so you could workaround it using something like (yes, looks ugly): $ printf "foo bar\n" | (while read a b; do printf "%s %s\n" $a $b; done; if test -n "$a$b"; then printf "%s %s\n" $a $b; fi) foo bar $ printf "foo bar" | (while read a b; do printf "%s %s\n" $a $b; done; if test -n "$a$b"; then printf "%s %s\n" $a $b; fi) foo bar From owner-freebsd-hackers@freebsd.org Mon Oct 22 13:42:22 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C7CDFFC70E for ; Mon, 22 Oct 2018 13:42:22 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 0A60A7A6B6; Mon, 22 Oct 2018 13:42:21 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id C22577B9; Mon, 22 Oct 2018 16:42:20 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? From: Lev Serebryakov To: Alan Somers Cc: "freebsd-hackers@freebsd.org" , cse.cem@gmail.com References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Mon, 22 Oct 2018 16:42:14 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="y8HNS5bptWoMgbaRyG5N1AanZym3cEBcL" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 13:42:22 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --y8HNS5bptWoMgbaRyG5N1AanZym3cEBcL Content-Type: multipart/mixed; boundary="iJZ9MacptV4o7GpeDwjqtxwbwzRbeQDam"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Alan Somers Cc: "freebsd-hackers@freebsd.org" , cse.cem@gmail.com Message-ID: Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> In-Reply-To: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> --iJZ9MacptV4o7GpeDwjqtxwbwzRbeQDam Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 22.10.2018 14:45, Lev Serebryakov wrote: > CURVNET_SET(so->so_vnet); > if (!SOLISTENING(so)) > error =3D so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, > top, control, flags, td); > else { > m_freem(top); > m_freem(control); > error =3D ENOTCONN; > } > CURVNET_RESTORE(); Oh, I'm idiot, it is eligible for TCO, if VNET is not compiled-in. --=20 // Lev Serebryakov --iJZ9MacptV4o7GpeDwjqtxwbwzRbeQDam-- --y8HNS5bptWoMgbaRyG5N1AanZym3cEBcL Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvN07ZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4/wpQ/+IraIkjYGi2fBmLa/C+4htMKS23C0QgmnlZVR/FV3WVPenGkoGZas4Zhh qLToKHsa4Qkczzyawn5Xm6bgIo5qyI0dxLtB8A+ZpsBsPwumjgHN3qIncG62fcmv 3FV7qZqtd2v34ex0X/VcQ9H2N9VP0BoNIi2yCg/ema2DsPiT+sogXqciI6iw4sgo SGbJoO2oRF7iHkXlQDLQQ3mIi5B36NViDs3n1N5rLMjIjprMZtMHa9QXjAx3Aq+r 4f+OBffpzPzV4rh39hTnTMZ0Cc7u8GfFEy5eWi8S1TQAPB6Ffzv7t+NGo44zWIpY 7RNKo3YYfoyaOfNHXtHc2FGidSsq9JUucn3V8B+Sqr0WVBhl4dExgeohhRg3Zzf0 wMUFw6GqS0nP/6Lmv3OsenuYABANOlpkXyKirSuuD8kz8fuL/qoscp7XArJ6A0Pl VR0xEaJXCpquFUSCzMkQm+2aPYAYxIiQkz8PXnDlOI8Oh45iZ2vTg5b1ObsojwzV 0Tz2Qpbkaqqi1zYsHuPDYsU19YaEUVDSqToEzIzewb1sKEsNdWeep5JmHO7WEz+f QbVHBmh27weizNnQqpkWjuiNpotEAfze6zZWm+4XTapduNJ9SVyWF70tGAw7/U/a ZPdLouoJYFbQRWZBA1b+o8q02keqixjolmPtWQ0gRCIzqEf/UTM= =KMqO -----END PGP SIGNATURE----- --y8HNS5bptWoMgbaRyG5N1AanZym3cEBcL-- From owner-freebsd-hackers@freebsd.org Mon Oct 22 14:35:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03D0FFFE2C9 for ; Mon, 22 Oct 2018 14:35:17 +0000 (UTC) (envelope-from mber2015@gmail.com) Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70C017CB57 for ; Mon, 22 Oct 2018 14:35:16 +0000 (UTC) (envelope-from mber2015@gmail.com) Received: by mail-wr1-x42c.google.com with SMTP id d10-v6so5645603wrs.5 for ; Mon, 22 Oct 2018 07:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=UY9OVMWfgST6Lc8VKtozK7cvFjm8HFhfwNJFzjSloIA=; b=IW6W+0Y9rrSti0XGjcE1VC1QblT5/+XMrawqrK+li8uSKsuWTa0hcy8wUlr6Q/RZVl 22xegqDMJcLWxKJD574ERRCF2CVs+yt9l94hEHTKxzovEQUcggzQDD0xfBdctMWG1L7U lY/41V6RsXJFYrJvJq29NG2Bokk96Y6g4BxzclvH+k15XvODmczrdAk6l5g7CRPmd2rH dAoIhtfyV6KRUCZTW7NMQSWaur1uSayWVjJsDVQq3gNT0uUYeFjLa7G3c76k4bms1iTY K+YSBt1YkFwFw3uKwIbcTVON7TD6ZPkdBwPt5LG1+EGzTOPWnYhbfGY5iPS7QtynVTao ydqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=UY9OVMWfgST6Lc8VKtozK7cvFjm8HFhfwNJFzjSloIA=; b=J2PZiOz6XUeo9hPN9INyGZ6Z9VqdzIpYuVoYdToyXdAl6x3YAYS2/YnljRafNVqVab 9GfOcq4Ug4JfrnhqsGKxVsur5EI1072sQlQcHYW3vH+/I5VYswWW8bF6JhXVgVRDKfWz vh4Ax7usVwTCxYCBZmlU6DmanS0slg/R4HZNdvTuGKanJN1VlYgvYmTOSiFH1TZf++4u QMDFERaSnlLsG0SAReVR8udrh/s5PbsE0Iq1MTMWa2HncfIh0qAKY5fs+FXNUynOzdHd 7Y0xu47p5hH4J7N5F2gjI7FbghOHjClS2tRAL9Dx6hgUQDudHMiXK5ejDCv6i43sqATo R7cw== X-Gm-Message-State: AGRZ1gIZGN56rUQ9C2QRVNfsn7LOibpe8lLPZxvYgW/AX9TgQUDmvPkF ADT5fgMwyrbu1HpBKW11NZWAHJBR X-Google-Smtp-Source: AJdET5eEm5SrQ27diPU0Idm0f8tjWGIDmNvamBJUGlpfUxmgCDWHZrFFn2J+z08HuRa86kb5ek25Qw== X-Received: by 2002:adf:de86:: with SMTP id w6-v6mr2475215wrl.295.1540218914671; Mon, 22 Oct 2018 07:35:14 -0700 (PDT) Received: from mb.tns.cz (pha.tns.cz. [84.242.109.107]) by smtp.gmail.com with ESMTPSA id q17-v6sm22266342wrw.19.2018.10.22.07.35.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Oct 2018 07:35:13 -0700 (PDT) Sender: Martin Beran Subject: Re: shell read built-in To: freebsd-hackers@freebsd.org References: <2fff355e-a729-d5d3-8199-f2bbba80c112@grosbein.net> <4ad1ea37-4714-a104-89a2-d7aef70d0f89@yuripv.net> <4cd38621-7387-6302-437e-b8fda3fbe1cd@yuripv.net> From: Martin Beran Message-ID: Date: Mon, 22 Oct 2018 16:35:11 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0.1 MIME-Version: 1.0 In-Reply-To: <4cd38621-7387-6302-437e-b8fda3fbe1cd@yuripv.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 14:35:17 -0000 On 10/22/18 1:32 PM, Yuri Pankov wrote: > BTW, it looks like last line is still parsed despite not having \n, so > you could workaround it using something like (yes, looks ugly): > > $ printf "foo bar\n" | (while read a b; do printf "%s %s\n" $a $b; done; > if test -n "$a$b"; then printf "%s %s\n" $a $b; fi) > foo bar > $ printf "foo bar" | (while read a b; do printf "%s %s\n" $a $b; done; > if test -n "$a$b"; then printf "%s %s\n" $a $b; fi) > foo bar If code in the while loop is more complex and you do not want to repeat it twice or define a shell function for it, you can use: while { read l; e=$?; [ $e = 0 ]; } || [ -n "$l" ]; do : any code that processes $l [ $e = 0 ] || break done; The condition tests that either read returns 0 (a line terminated by '\n') or puts a nonempty value to $l (the last line not terminated by '\n'). The break command eliminates further read after EOF (nonzero return), because on a terminal, after EOF (pressing ^D), read returns false once. When called again, it continues reading from the terminal until the next ^D. -- Martin Beran From owner-freebsd-hackers@freebsd.org Mon Oct 22 16:10:40 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0F7E1036E6A for ; Mon, 22 Oct 2018 16:10:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 585678202E for ; Mon, 22 Oct 2018 16:10:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f174.google.com with SMTP id o14-v6so37541976ljj.2 for ; Mon, 22 Oct 2018 09:10:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=K95IyqVFzIXs4WniWeTbo+xIYTUuSaNPXMAhqMRT54k=; b=n8m6lGjgoaSeGCNBzNhKHRWn7kzE7W/j/Nxz1Ur3m30FO1lnMwffgbU/NAoWXT66SW qwzmuPkd/9dV9NYjt4ZFZ2JZHmBIpdsNcdikMCiYFq666BX/ROz77du1yF0vO+IUP2ka uBy1xNXQCi2kgRHhsnBHG64Q8RS1ZFd1NzdG8hynoF58uO8BkWm/FwP/VMRWRbMBgTdF FBDGLaCjrXW0iXFQzVT8NoQZy+1AJ6Mf+o2igYFK/4Qf0EF/opDcIliS18EZ7xrwbs+i MeTv30uV9h2WDtUXQgUF8/NR9GrDLWsj4NG9Iz1zjFizLwUcoGeh52zAYtTBkyM14xc0 Tvgw== X-Gm-Message-State: ABuFfog5iD8WeUNK9ayHXKeA5gywiB4djdp5ztVUubULIsSawmV/Iz5+ ei3lOk9HQ6zXdLou55iBLW61pVKTsmpGdPLI2GCSY0IZ X-Google-Smtp-Source: ACcGV62QFssVVrZ3u6aUOc0GqjU4rGNyWE2ZV3mGzswYvOFaaCrNN6dbCcBHgPtQXdBgGM3iRNPvNUexIlbcZ4Cc45I= X-Received: by 2002:a2e:544f:: with SMTP id y15-v6mr32273576ljd.51.1540224191501; Mon, 22 Oct 2018 09:03:11 -0700 (PDT) MIME-Version: 1.0 From: Alan Somers Date: Mon, 22 Oct 2018 10:02:59 -0600 Message-ID: Subject: SNIA SDC 2018 recap To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 16:10:41 -0000 The SNIA Storage Developers' Conference was held in Santa Clara during the last week of September. Jim Harris, John Hixon, Nick Principe, Michael Dexter, and myself attended. As far as FreeBSD goes, here's a summary of the juiciest bits: NVDIMM/PMEM: A lot of companies are still pushing persistent memory products. They're getting better, but still quite vendor-specific. Fortunately, there are standardization efforts in place. JEDEC is standardizing the hardware (NVDIMM-N, NVDIMM-P, NVDIMM-F). Every major memory company (but not CPU company) is on-board. SNIA is also trying to standardize a programming model (but not the precise API). Windows and Linux currently support it, with differences. There will probably be some additional changes to the model. https://static.ptbl.co/static/attachments/187585/1537988510.pdf?1537988510 . iX Systems reported some impressive benchmarks using an NVDIMM as a ZFS slog device. Several databases are adding pmem support. A few filesystems have some level of NVDIMM support, and the NOVA filesystem is being written from the ground up to take full advantage of NVDIMM. For example, directories are stored as in-memory data structures that never get serialized. The lesson here is that FreeBSD needs to support the standard NVDIMM programming model too. OpenChannel SSDs: These are SSDs that expose more of their internal implementation details to the host. Specifically, they rely on the host for at least part of garbage collection. They also expose their multiple internal busses to the host, so it can choose how to stripe data across them. Overall, the programming model is surprisingly similar to that of SMR hard drives. Unfortunately, the standard is a bit murky. Different speakers could not even agree on whether there is a standard. This is the best presentation on the topic: https://static.ptbl.co/static/attachments/187321/1537829929.pdf?1537829929 and this is the closest thing there is to a standard ATM: https://openchannelssd.readthedocs.io/en/latest/ . The lesson here is that FreeBSD needs to plumb these devices' properties up to userland and perhaps expose them in zonectl(8) (easy) and add filesystem support (very hard). NVMe: If there were an award for most popular buzzword, it would've gone to "NVMe". Everybody and their mother had something to say about it. But I personally paid little attention (except as regards OpenChannel). Seagate dual-actuator hard drives: Seagate is coming out with hard drives that pack two servos into a single case. Each servo can access half of the platters. The drive reports each servo as a separate LUN to the host. There is little FreeBSD needs to do here. To make zfsd(8) work correctly, we should add lun info to the drives' physical path strings. And it might be nice if zpool(8) prevented the user from adding both LUNs of the same physical drive to the same RAID group. But that's arguably out of our domain. SPDK: The storage-plane developer's kit is like Intel's version of Netmap, but for storage. It's a say for userland programs to access storage devices directly, bypassing the kernel. The benefits are negligible for spinning media, but can be significant for fast NVMe drives. SPDK has multiple backends for different I/O controllers, including some that are kernel-based. Notably lacking is a POSIX AIO backend. That's probably the biggest gap in its FreeBSD support. iX Systems wrote a blog post about the conference, too. It covers Swordfish and Samba, two topics I ignored. https://www.ixsystems.com/blog/snia-sdc-2018/ -Alan From owner-freebsd-hackers@freebsd.org Mon Oct 22 16:23:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 767C8103765C for ; Mon, 22 Oct 2018 16:23:38 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC9A482C15; Mon, 22 Oct 2018 16:23:37 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: by mail-wr1-x42f.google.com with SMTP id q7-v6so17806126wrr.8; Mon, 22 Oct 2018 09:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VDML47pm5YM7SCfXD3NumVvOhzKVXqRsD0XHjhi7Aws=; b=qs5RDFmbVWXg5yE+6fpjomelhtau2+9RAhkhoZdxP81pxegQzhRkOIVnQgO2wZF8kw q2Silp4br5NpZ+F0R2Jz7UAzEIgMrsmXf2d9JeInbtINP/bxsUw8LJYJ4Yf5eOTXvfvn CHylOzuZtjhOm3w5MN1WXyhaKAjDjYtluMi8psI8SKX5nyzl16IPzB1ViY7zkpoC/kyB cuxtg/arkrTdn/DmR9MhGmGkUZ7OZhIFuu7IS7RLN+JPZapXTQrITcDMYIkGVimGIkhy paJLmXm3R0ihkqeOLp4Q/fgEgRItOXNo1lfwTXx2m3fnKe74qWYrMFPyNCjDGg4+DF2U gY0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VDML47pm5YM7SCfXD3NumVvOhzKVXqRsD0XHjhi7Aws=; b=TvRd+wo6dlyQqRSMoiAgn0TFvQCLoUbFaipE7/ATlPM4sen2dvBMWWf79NBXGNy4rq F66tQyhJxHe1xhwnNUkCR6JDMkN3GL1vxc1Sy5J3I2tspwO1kMEeoap4zWez0hi1SR+t zo8rkwo0U1RpLIxG8vDTNTPK8dd36KSViGyCCZm7jwjHvGH9ydr42hjdIjpnBMNxpgLO GIrTn+PS67q+RyelO2mV4S8j1gUweibVBE0ODvkRmXODTgu8z7RbGN09JT2vmXHw/kD9 fUzDsM3Z8P9tzOm+Fvj4pm70TEtDr+gOaFuAHvciuvtDHgPtRqRKaizGWpRpMeLyimPQ 7gJw== X-Gm-Message-State: AGRZ1gLyoDp17X2y3SfEbym1fFDxRjG0Uh6wwPIG8fw3vw52xwME+1le WzaSEMR/fIxqT6cMbfik72RdrwQuOluZZFYLHi4sj73O X-Google-Smtp-Source: AJdET5dOmC+v0P7Hu3ahyBQPtX5RAoQ4SUu2PFe1o+4cw2E/6JWLJmoJnsTbeAB2ePUm2INbFiI9RJF97Q5L8VNN6lQ= X-Received: by 2002:a05:6000:108d:: with SMTP id y13mr6143571wrw.226.1540225416278; Mon, 22 Oct 2018 09:23:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vijay Singh Date: Mon, 22 Oct 2018 09:23:17 -0700 Message-ID: Subject: Re: SNIA SDC 2018 recap To: Alan Somers Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 16:23:38 -0000 I was there as well Alan :) On Mon, Oct 22, 2018, 9:13 AM Alan Somers wrote: > The SNIA Storage Developers' Conference was held in Santa Clara during the > last week of September. Jim Harris, John Hixon, Nick Principe, Michael > Dexter, and myself attended. As far as FreeBSD goes, here's a summary of > the juiciest bits: > > NVDIMM/PMEM: A lot of companies are still pushing persistent memory > products. They're getting better, but still quite vendor-specific. > Fortunately, there are standardization efforts in place. JEDEC is > standardizing the hardware (NVDIMM-N, NVDIMM-P, NVDIMM-F). Every major > memory company (but not CPU company) is on-board. SNIA is also trying to > standardize a programming model (but not the precise API). Windows and > Linux currently support it, with differences. There will probably be some > additional changes to the model. > https://static.ptbl.co/static/attachments/187585/1537988510.pdf?1537988510 > . iX Systems reported some impressive benchmarks using an NVDIMM as a ZFS > slog device. Several databases are adding pmem support. A few filesystems > have some level of NVDIMM support, and the NOVA filesystem is being written > from the ground up to take full advantage of NVDIMM. For example, > directories are stored as in-memory data structures that never get > serialized. The lesson here is that FreeBSD needs to support the standard > NVDIMM programming model too. > > OpenChannel SSDs: These are SSDs that expose more of their internal > implementation details to the host. Specifically, they rely on the host > for at least part of garbage collection. They also expose their multiple > internal busses to the host, so it can choose how to stripe data across > them. Overall, the programming model is surprisingly similar to that of > SMR hard drives. Unfortunately, the standard is a bit murky. Different > speakers could not even agree on whether there is a standard. This is the > best presentation on the topic: > https://static.ptbl.co/static/attachments/187321/1537829929.pdf?1537829929 > and this is the closest thing there is to a standard ATM: > https://openchannelssd.readthedocs.io/en/latest/ . The lesson here is > that > FreeBSD needs to plumb these devices' properties up to userland and perhaps > expose them in zonectl(8) (easy) and add filesystem support (very hard). > > NVMe: If there were an award for most popular buzzword, it would've gone to > "NVMe". Everybody and their mother had something to say about it. But I > personally paid little attention (except as regards OpenChannel). > > Seagate dual-actuator hard drives: Seagate is coming out with hard drives > that pack two servos into a single case. Each servo can access half of the > platters. The drive reports each servo as a separate LUN to the host. > There is little FreeBSD needs to do here. To make zfsd(8) work correctly, > we should add lun info to the drives' physical path strings. And it might > be nice if zpool(8) prevented the user from adding both LUNs of the same > physical drive to the same RAID group. But that's arguably out of our > domain. > > SPDK: The storage-plane developer's kit is like Intel's version of Netmap, > but for storage. It's a say for userland programs to access storage > devices directly, bypassing the kernel. The benefits are negligible for > spinning media, but can be significant for fast NVMe drives. SPDK has > multiple backends for different I/O controllers, including some that are > kernel-based. Notably lacking is a POSIX AIO backend. That's probably the > biggest gap in its FreeBSD support. > > iX Systems wrote a blog post about the conference, too. It covers > Swordfish and Samba, two topics I ignored. > > https://www.ixsystems.com/blog/snia-sdc-2018/ > > -Alan > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Mon Oct 22 16:34:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6396D1037F5A for ; Mon, 22 Oct 2018 16:34:01 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D295783845 for ; Mon, 22 Oct 2018 16:34:00 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f170.google.com with SMTP id k11-v6so5609545lja.5 for ; Mon, 22 Oct 2018 09:34:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ncyO88DYcWPeLXByMJnzvoJteXYIxk1EBf6XXfRLYMA=; b=RUf+7BjxiukvrTayKVUzFYUR1XulNrCc7hWdfbOPT2NvIyYa0wKwCYNYoCUqcSScOm 7D6rzTVF8pnYkKt22YQEmgAkX8sqh/oCCAujVc7d5jfMig3JnixjXhHUtTRG1SYucPlC 89HUX9N5EEz/BmZaoNecE/OKmbtbHYWDP3OVRU8WAlEHL+2vaImp3kTXOzl5KP0imSUk uwLVoPaHNfb9NlcZgTXFN4gY861O4Eg2TGZ3KNyUVMvkZxD8AGXp0EDC1xVqRCU/SyZZ WGlSYYOdNIYfqJEwVZ8lBqv6biyf4aeZhQu9Nw7djtvykNPlHzl1pQnflnUCcSUizEYb Q0Og== X-Gm-Message-State: ABuFfogVLVXfBWpMwJjmlCvl0u2rKCysVXN5S4ALlttepHGdEW4b6coY +PTk5Q5U6/8wThNVr8apfmjEEOtQNI1WQyJ+sFU= X-Google-Smtp-Source: ACcGV62Q4tYiJZEQEv710w6XnO2TGpDCiAHBPpC16cmvlaCOwAsO3ps3yk41asiN5ulzpFzFuUZjazR4GLFMm9lfwak= X-Received: by 2002:a2e:2b08:: with SMTP id q8-v6mr30967480lje.128.1540225728197; Mon, 22 Oct 2018 09:28:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Mon, 22 Oct 2018 10:28:35 -0600 Message-ID: Subject: Re: SNIA SDC 2018 recap To: vijju.singh@gmail.com Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 16:34:01 -0000 Ahh, I didn't notice. Were you associated with your employer? -Alan On Mon, Oct 22, 2018 at 10:23 AM Vijay Singh wrote: > I was there as well Alan :) > > On Mon, Oct 22, 2018, 9:13 AM Alan Somers wrote: > >> The SNIA Storage Developers' Conference was held in Santa Clara during the >> last week of September. Jim Harris, John Hixon, Nick Principe, Michael >> Dexter, and myself attended. As far as FreeBSD goes, here's a summary of >> the juiciest bits: >> >> NVDIMM/PMEM: A lot of companies are still pushing persistent memory >> products. They're getting better, but still quite vendor-specific. >> Fortunately, there are standardization efforts in place. JEDEC is >> standardizing the hardware (NVDIMM-N, NVDIMM-P, NVDIMM-F). Every major >> memory company (but not CPU company) is on-board. SNIA is also trying to >> standardize a programming model (but not the precise API). Windows and >> Linux currently support it, with differences. There will probably be some >> additional changes to the model. >> https://static.ptbl.co/static/attachments/187585/1537988510.pdf?1537988510 >> . iX Systems reported some impressive benchmarks using an NVDIMM as a ZFS >> slog device. Several databases are adding pmem support. A few >> filesystems >> have some level of NVDIMM support, and the NOVA filesystem is being >> written >> from the ground up to take full advantage of NVDIMM. For example, >> directories are stored as in-memory data structures that never get >> serialized. The lesson here is that FreeBSD needs to support the standard >> NVDIMM programming model too. >> >> OpenChannel SSDs: These are SSDs that expose more of their internal >> implementation details to the host. Specifically, they rely on the host >> for at least part of garbage collection. They also expose their multiple >> internal busses to the host, so it can choose how to stripe data across >> them. Overall, the programming model is surprisingly similar to that of >> SMR hard drives. Unfortunately, the standard is a bit murky. Different >> speakers could not even agree on whether there is a standard. This is the >> best presentation on the topic: >> https://static.ptbl.co/static/attachments/187321/1537829929.pdf?1537829929 >> and this is the closest thing there is to a standard ATM: >> https://openchannelssd.readthedocs.io/en/latest/ . The lesson here is >> that >> FreeBSD needs to plumb these devices' properties up to userland and >> perhaps >> expose them in zonectl(8) (easy) and add filesystem support (very hard). >> >> NVMe: If there were an award for most popular buzzword, it would've gone >> to >> "NVMe". Everybody and their mother had something to say about it. But I >> personally paid little attention (except as regards OpenChannel). >> >> Seagate dual-actuator hard drives: Seagate is coming out with hard drives >> that pack two servos into a single case. Each servo can access half of >> the >> platters. The drive reports each servo as a separate LUN to the host. >> There is little FreeBSD needs to do here. To make zfsd(8) work correctly, >> we should add lun info to the drives' physical path strings. And it might >> be nice if zpool(8) prevented the user from adding both LUNs of the same >> physical drive to the same RAID group. But that's arguably out of our >> domain. >> >> SPDK: The storage-plane developer's kit is like Intel's version of Netmap, >> but for storage. It's a say for userland programs to access storage >> devices directly, bypassing the kernel. The benefits are negligible for >> spinning media, but can be significant for fast NVMe drives. SPDK has >> multiple backends for different I/O controllers, including some that are >> kernel-based. Notably lacking is a POSIX AIO backend. That's probably >> the >> biggest gap in its FreeBSD support. >> >> iX Systems wrote a blog post about the conference, too. It covers >> Swordfish and Samba, two topics I ignored. >> >> https://www.ixsystems.com/blog/snia-sdc-2018/ >> >> -Alan >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> > From owner-freebsd-hackers@freebsd.org Mon Oct 22 20:50:43 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32101106DEC2 for ; Mon, 22 Oct 2018 20:50:43 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9ABBD72F47; Mon, 22 Oct 2018 20:50:42 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-lj1-x229.google.com with SMTP id j4-v6so38293230ljc.12; Mon, 22 Oct 2018 13:50:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oxTM1Kx5qKua6KqkCMXu4tVLo9SYPWhwSOmRWFuEQ3Y=; b=gpYV6/Z+hh7W66Zp8Y9hAf/mDAAujdOZHcBtFSEkBL/bQGsDbz6ocOzaO8bZVe8zuD uLhaPilDMkb+ip4cGP8dFAS1q0vDFTZmHtHEDngJGwTgwkfojYZdJL5SMFQKLDXyJtAX FYMwXziogXme+hpmJv1GqOAAQCZuUAEtR62JxBk4bPkVOuYcO923tvAu7GNkcZcmaAe8 9qTaDgguaeFrkKXd7vrs4ITRb+zsqHb2zua6O8GvU3+UtpR/j3wcF2ZTlS6edNZIB4Dz bNv6KcQMjLgjxdCexuNha4R7aFcdiZXFiSc2BkovYY5WqKXyTDv/sGiJO/hvNMnOp2ZS 8CdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oxTM1Kx5qKua6KqkCMXu4tVLo9SYPWhwSOmRWFuEQ3Y=; b=C6BHA9hmOMCcQ91bDhDfISqkojCH7ffReStIy0bVLybPT97MZP2LTwVLq1qRN2J0iB MPYxdxHjZtQCRjzsAe9PDQ6aTdXeKFcWBQ4Rjb+AgNE/G+Sz8sDuLpLrge7ydTiryred Fut7fFtt0YnNtW1RJdlcnnUq2HjFYXhCMvdWL/mPxdJ66KnjJ027aW0kmLLvqyfnoSJT V1BQbuc/BaUCuGaMhHMeeahvPd1n7NDJ1xQQ8GhYOtCDO7NBJ7Wtb/FlnzRCBI5PNYl/ U3tFU2CAjvCJgQlibCJtWEL4i7y3/fPZra5LBnVCtt+WiIeq6gKMuLs5Usan/OLq5Dtt Mltw== X-Gm-Message-State: AGRZ1gKcO8TwB2XMhik2O/qAhTfWzmP8QiJ4ReJLuOl7VNl+w+uElmsb 9TIpacXO09ZmiaW0JB/BR1AQ7bKibjBoxZ+aMFnIhg== X-Google-Smtp-Source: AJdET5fKpqnTudxbJdSiB1/jww/EXrd06r9exoNFqH/ZFrz46Xjne9bWL96FsqkDpwJT3Ct2owNzNjJM8VxWtTjOvp8= X-Received: by 2002:a2e:85da:: with SMTP id h26-v6mr42028ljj.122.1540241440997; Mon, 22 Oct 2018 13:50:40 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> In-Reply-To: From: Ryan Stone Date: Mon, 22 Oct 2018 16:50:29 -0400 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: lev@freebsd.org Cc: Alan Somers , freebsd-hackers@freebsd.org, cse.cem@gmail.com Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 20:50:43 -0000 Adding -fno-optimize-sibling-calls to the compiler flags will eliminate the TCO. On Mon, Oct 22, 2018 at 9:44 AM Lev Serebryakov wrote: > > On 22.10.2018 14:45, Lev Serebryakov wrote: > > > CURVNET_SET(so->so_vnet); > > if (!SOLISTENING(so)) > > error = so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, > > top, control, flags, td); > > else { > > m_freem(top); > > m_freem(control); > > error = ENOTCONN; > > } > > CURVNET_RESTORE(); > Oh, I'm idiot, it is eligible for TCO, if VNET is not compiled-in. > > > -- > // Lev Serebryakov > From owner-freebsd-hackers@freebsd.org Tue Oct 23 11:46:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB1F3107661F for ; Tue, 23 Oct 2018 11:46:35 +0000 (UTC) (envelope-from Shreyank.Amartya@amd.com) Received: from NAM05-DM3-obe.outbound.protection.outlook.com (mail-dm3nam05on0612.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe51::612]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70B4D77693; Tue, 23 Oct 2018 11:46:35 +0000 (UTC) (envelope-from Shreyank.Amartya@amd.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ykuh21WZglDvZH7EvTN6qLcMWEyiVUMPd09z2+yqlf8=; b=K9p348xXQUeQTWSPMeaZUFAoX95i2rBkTVDfaDf5Wchy9DC/ALeckMmckwJZK4Fn+8I4dGyxFo2Oqtf3p2yoxq7sWmGnxAFWokf9PvcOe2t7kiAZaPEBVlXE9peVtfzJerYjX0wcgmPf4s80ShyZ1MHjfHbhP5MnZX1BtefEapI= Received: from SN6PR12MB2813.namprd12.prod.outlook.com (52.135.100.27) by SN6PR12MB2813.namprd12.prod.outlook.com (52.135.100.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1250.22; Tue, 23 Oct 2018 11:46:33 +0000 Received: from SN6PR12MB2813.namprd12.prod.outlook.com ([fe80::f43e:fb06:f5cb:86b6]) by SN6PR12MB2813.namprd12.prod.outlook.com ([fe80::f43e:fb06:f5cb:86b6%6]) with mapi id 15.20.1250.028; Tue, 23 Oct 2018 11:46:33 +0000 From: "Amartya, Shreyank" To: "freebsd-hackers@freebsd.org" , "marius@FreeBSD.org" , "mav@FreeBSD.org" Subject: FW: eMMC AMD platform Thread-Topic: eMMC AMD platform Thread-Index: AdRqp/Txi+uhfRAQRZex7esIDeBosgAHbQcQ Date: Tue, 23 Oct 2018 11:46:33 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Shreyank.Amartya@amd.com; x-originating-ip: [202.56.249.162] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; SN6PR12MB2813; 20:MloxfhHMRJTEbrNvrn0/Gyz3O4Vebtn39O8N+40IEMoRKxTptS16rnWuvOs/fwYZn8THJqlVfelYzUeIWh6v9mwBjZyIR3mUvSCBk9bvvOpTaz/GrekF1JIarifsLYe5nuX4nmHNpZzG7/yqcw1mqgAh2JitBD85qSJ+ujs2ykkXi7xN9e9Mzh++iGhOTTneaB3C3GQFh4HJra+5oM6OHOg3s22CnfMPy59Pm3t5TQsbArnHOxsGMgSfXMIeNxf9 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 46c9dbcb-a20f-4722-f0fb-08d638dd2e54 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:SN6PR12MB2813; x-ms-traffictypediagnostic: SN6PR12MB2813: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(21748063052155)(28532068793085)(190501279198761)(227612066756510); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(3231355)(944501410)(52105095)(10201501046)(93006095)(93001095)(6055026)(148016)(149066)(150057)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123560045)(20161123558120)(20161123564045)(201708071742011)(7699051)(76991095); SRVR:SN6PR12MB2813; BCL:0; PCL:0; RULEID:; SRVR:SN6PR12MB2813; x-forefront-prvs: 0834BAF534 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(376002)(136003)(39860400002)(396003)(199004)(189003)(81156014)(2940100002)(236005)(105586002)(102836004)(5660300001)(966005)(81166006)(8676002)(72206003)(478600001)(8936002)(6306002)(54896002)(9686003)(9326002)(7736002)(71190400001)(97736004)(25786009)(71200400001)(14454004)(33656002)(6436002)(106356001)(68736007)(7116003)(55016002)(74316002)(2900100001)(606006)(2473003)(790700001)(99286004)(446003)(110136005)(86362001)(316002)(486006)(53936002)(3846002)(11346002)(450100002)(476003)(6116002)(5250100002)(229853002)(2906002)(2201001)(2501003)(6506007)(26005)(93156006)(66066001)(14444005)(7696005)(76176011)(186003)(256004); DIR:OUT; SFP:1101; SCL:1; SRVR:SN6PR12MB2813; H:SN6PR12MB2813.namprd12.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: amd.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: gAI09TcDBbV9PbEtEpmlWD84ZajzGyfET6NKeYjYAczjB0OnOYGraKBuRtZNoXlNdgOOuu6dv5ek6/pJcgBPQH4w27dVGxMTQHvD2yVINdLL9T3KcbIfiLMi2TnJWmMaJvMaiKV3rV8cE9Gqibnzes99II6zjnoV+OUIb7Qi29AouNzDYZLAKElguwACQP/B4VKz7/Fn551SSdrTlygoR62xDBpxaJUkzu/S2YIk1iyRG7TwI92D3zMdiT2UVRF+ZV41qOZaNWbWntJnN2Z56rwIrHwP8n2DxvV6xsLwEQ1kwxMNvFze5gEa6A8nR+bCOOxVNGuXzRNTccRwPgdwS3bJBwNIfIgX5SzN+CEt/uo= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46c9dbcb-a20f-4722-f0fb-08d638dd2e54 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Oct 2018 11:46:33.4139 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR12MB2813 X-Mailman-Approved-At: Tue, 23 Oct 2018 12:39:49 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 11:46:36 -0000 Hi, I have an AMD eMMC 5.0 controller connected to an eMMC device. I recently got around to enabling HS400 mode but the performance is still s= ame what I was getting in HS200 mode. Is there a way to verify the HS400 mode is active? Patch for enabling HS400: https://reviews.freebsd.org/D17644 In order to enable the HS400 mode I had to clear the Sampling clock select = bit (Host control 2 register) and then set it later (as part of the patch). Now, I suspect that this led to reset of the tuning circuit and that even t= hough HS400 mode is enabled, since it is not tuned, I do not see increase i= n performance. Could that be a possibility? There is a similar patch for linux and I've tried it, it works on my board = with HS400 mode (~200 MB/s), measured using iozone. Linux patch: https://patchwork.kernel.org/patch/10086747/ Thanks Shreyank Amartya From owner-freebsd-hackers@freebsd.org Tue Oct 23 13:26:56 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52D84FD7DC5 for ; Tue, 23 Oct 2018 13:26:56 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id A85857C363 for ; Tue, 23 Oct 2018 13:26:55 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 7D5A3AA6; Tue, 23 Oct 2018 16:26:49 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Ryan Stone Cc: freebsd-hackers@freebsd.org References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> Date: Tue, 23 Oct 2018 16:26:41 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="TBzVzW1niMUhHplfKbuanMsOfPGfwxkNK" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 13:26:56 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --TBzVzW1niMUhHplfKbuanMsOfPGfwxkNK Content-Type: multipart/mixed; boundary="4ZnTLQATgwaJPyMTWhLp0DW4j7CQmgB8L"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Ryan Stone Cc: freebsd-hackers@freebsd.org Message-ID: <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> In-Reply-To: --4ZnTLQATgwaJPyMTWhLp0DW4j7CQmgB8L Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 22.10.2018 23:50, Ryan Stone wrote: > Adding -fno-optimize-sibling-calls to the compiler flags will eliminate= the TCO. How could I add it for kernel properly?! Adding "CFLAGS=3D-fno-optimize-sibling-calls" to make.conf affects modules build= , but not kernel itself (I'm surprised, to be honest). --=20 // Lev Serebryakov --4ZnTLQATgwaJPyMTWhLp0DW4j7CQmgB8L-- --TBzVzW1niMUhHplfKbuanMsOfPGfwxkNK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvPIZFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R49Ekg/5Af0mNVcISW0qkaVfRorKq7k325x+lwLTUYbhVfpTlTKAxrEPyZl9s8Bn +7gQPSuqgXXFDy4H+WSogNml2NZW6IPHcOJryWdgu4Sx3a3VtVmwI5szpN/GiFU/ nVpc4KvUs/0sv59LsFjjkxULwjilCK7eb9RRGky5gMCRvTS4ukRTJCf2i+rBVvg1 uDDQSec+IKvtqBqdcR3sGAgBzg8RCOQ2CRfziS3EobRHOBjJw5u7NDswhd0LC8TY l/6hJsMUxZPUwzeIeeF5uCuHpRyGTw6vpM3RWC7wetHgq/1SdVgHmUnM1YBmgrjz UECpAsnlsmdDK1Ae405ZnF6YI9PT7+0XGrhm4AQiYzT0YmNfjMudNQHtfoyTKMKN GhuumzAGV+bw8ogxt1qTrIynZjoSE91naBzcdPeegwiyP/mOuPAs0q7eT165FVjo Vh4Ym1bDjQra9N/b/85TO6XnpNdqZoSrIyvPnQNsdBdpjCINc/l1dqZWPOeOwz3m +jqWWlkx5gFODSeM7R8Mp5BQxWPVAN9oguoCnafQb+cPjEHap6lV/sB3gQ+b3fl+ W4MurHBLsE7j1lwGLfnlaNoVXUccOkPkqoq2hTum+dsUuMzEx2gyA3Apex0aBlQc WgAhvRSx41IP4TavvHihpBM/gA1ifmdRdA8gdOqJWxb9eGZV2r8= =GLNf -----END PGP SIGNATURE----- --TBzVzW1niMUhHplfKbuanMsOfPGfwxkNK-- From owner-freebsd-hackers@freebsd.org Tue Oct 23 13:48:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E727FD8B5E for ; Tue, 23 Oct 2018 13:48:26 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DF85C7D47D; Tue, 23 Oct 2018 13:48:25 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id E173D20D56; Tue, 23 Oct 2018 09:48:19 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Tue, 23 Oct 2018 09:48:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= subject:to:cc:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm3; bh=O 8r6R5kUwIURSe4jCDjIvi2mNm/SnTVR3CqXBC/Lxo4=; b=L3nEKYDgGstuTfEfU jRTvAmwg724Q182oZvEKt12EE8q9nxaHWqz2flUB1p9mfXuO1rP7TDv+Dexu0hKo OIoDPiAx2hGIzBGm6eIsFRaURX5nNUCIQh6DtSEuj18ilTopSx/SycnkVPB357yQ ZwN8zkRMYq8zEKyZ4F6MdDEs6/cmAvq5K9qC2QHA6F2k+9EL0XhW/nB00FZIY9m0 9bsAPdUwzfWYCOSbzP6WIHJK28PKI2PFrfmUqbgrgJc7yRfzGRfMg8t/CnpHE7og 0WObDK6Ec94fuBHJ923HjibpTHt/YCBEdF6HClRzmaiPEDWf4/PhcV7W6vdTJBSB kpJCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=O8r6R5kUwIURSe4jCDjIvi2mNm/SnTVR3CqXBC/Lx o4=; b=HQy/5G2YbZVyttqQVCZRmn4dmHtgpoQOaQQNNaHvxK/Xg2id25izSOubU I4y9HjbcljKcXC0El0niXMzTKCINIM+/tpCsgHWJgUmt23pv2fZpJHP/y2kG8TL7 WeBnqeC4gbdqhJW4he7gfKwtcCeaUK+OZbltvQw0ffPzP2no0LffWek1KHNeeSOZ I4C1PDOIs8ZsBX6DC1gWE4nYrbpeKwtkhqorjAmtBanooU12ovsJuPAPIFG1VvPL kQE/JVzzX5xsTzyxgbZtdlfoRdVWoEpB1YFun/UZXkXfrlzu3TNkW8eaPFigoeQh ySAfgkMo01hKMdKGB3iQd0OgBBqig== X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [62.183.124.190]) by mail.messagingengine.com (Postfix) with ESMTPA id DB74B102DE; Tue, 23 Oct 2018 09:48:17 -0400 (EDT) Subject: Re: What is wrong with dtrace's stack()? To: lev@FreeBSD.org, Ryan Stone Cc: freebsd-hackers@freebsd.org References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> From: Yuri Pankov Openpgp: preference=signencrypt Autocrypt: addr=yuripv@yuripv.net; keydata= xsBNBFu8u6IBCADB11gP0QwnorrHjqAtKLHKHNHskhy0s7jqJKfx0YqXgVBKGLJ9/mjLAz0F CBNvemHSDDTs0mEZ9cBKKi6cmsav6+UQgr//yai6hvXLBJqKchSFO4MhmdvBtsGFq1yKz5Zi uhjmimKyIpgBgvMdbgGbGq6cnSB2uEPmZuJr419SVRODOkXukU+F5WHgaHzDdHAIu1asCt2B +6msxqIqlFWcXyZyTGicTGGvC/PFIsVRUtD1dIJANTC876g7DTb7LZXWiWwJpSJ4GKMXMHVX Ct9BoQ4i3nhKbOxb6Io1wsy+NFyWsTJ9KYrxKKPJP3oG8BWb/cqlFqnE4eNSsiq2q7krABEB AAHNIFl1cmkgUGFua292IDx5dXJpcHZARnJlZUJTRC5vcmc+wsCUBBMBCgA+FiEE+Gq3PsPe LT4tL/9wk4vgf7Eq4WwFAlu9Cn0CGwMFCQWjmoAFCwkIBwMFFQoJCAsFFgMCAQACHgECF4AA CgkQk4vgf7Eq4WxuPQf9HccaDyusO1J+wDQNlp9/uU0cnIfjHAeG80xrAfN9Vnf1wO9T2/WI iYlIdK+KVnhSa/DeBuHq/asfpUbrOleTF0hzG39os+95DzuT9a/j5XeQGuBgNbpVB+10zR3I 5AagSQetHilcZtz65g9GTUuIxb+xDaBehFBjyYXApfNE6yY5IlzDZpM7MOOLLFm2mQwQ8yjS eZ4jA6qW6/QMXRTkmpC9EXIeWDuNgWBwszaFGR6oUIpl0mGmwdJkEKwUazt6OuoDilMNZefZ 0pVFZBhnE46vK+6FDDFZE3BkeHVnqvy2QGL/6uKhSHc0lChCEPHnhqz6v23MwcQ6ktVWzvBJ oM7ATQRbvLuiAQgAyood0Pd96wzY+GQPBYQUNkZZgYL8Di3AzyC94dFe4d/Mt/h4rIBUnFwA g7Ha05WGdW0V5A/RRxDcpwXL9Jf97hiQ5PI2hiAxNEz/DkAUafiGlPfwR5wKqysUyRiKJQ2o ctpvssdsoXXOgeLo1jA6ghda1jg/spjlsPlS5ZTpKx3GWuTybV/VDhmwKWZfGUzPBJeAgDTf BdW4PTFs1IvvC2KBlhnPgcLBUtTlAdXOEj4DLuXw+Fn7K/ckZdOn3aRANmE+wf4+f+UUgtLB NmbP7ZifyUX5RyddsnI+fZmtsUDHxCReNIWQ6TBUJmb21aoBIN6HEHJbY28ZSCmf5owuMwAR AQABwsB8BBgBCgAmFiEE+Gq3PsPeLT4tL/9wk4vgf7Eq4WwFAlu8u6ICGwwFCQWjmoAACgkQ k4vgf7Eq4WyA3AgAqgGTHKMVAS2WuNGuW9uI+YtY6ZbwmGG94fkOZbefgRSfO5Am+HSblA95 IdotvQa8VkFmvVjbnvaM8XmJG5H17m0GF3sVaJUbJ4euDnRrBPCr6KwRQQd83Svxkbdicvo7 J031FrkJZW8zD9DH4QgzJNTKPFrwx9v3DhD/8iyn9tGvnHepy7O24nY5hl6PacrgSgLVeir/ lUbueAC/gP1AWLv3gdw7b83J7rftWauimj/vpFMD8CDSyJNODgQ8DdM0TU4qjABWGMs9r2Rw QehNbYf5f/2QuW/Q5NGaRSNW2HS/cpp62XtTKmxj5wwk6EMbtNE/6WQpumfdmK2UGLjcJQ== Message-ID: <27f7681e-7870-ae21-0563-2f31e58d1603@yuripv.net> Date: Tue, 23 Oct 2018 16:48:15 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 13:48:26 -0000 Lev Serebryakov wrote: > On 22.10.2018 23:50, Ryan Stone wrote: > >> Adding -fno-optimize-sibling-calls to the compiler flags will eliminate the TCO. > How could I add it for kernel properly?! Adding > "CFLAGS=-fno-optimize-sibling-calls" to make.conf affects modules build, > but not kernel itself (I'm surprised, to be honest). Try using: COPTFLAGS+= -fno-optimize-sibling-calls From owner-freebsd-hackers@freebsd.org Tue Oct 23 13:50:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59DD0FD8C9E for ; Tue, 23 Oct 2018 13:50:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id C8FCC7D6BF for ; Tue, 23 Oct 2018 13:50:12 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 0ADF6AB3; Tue, 23 Oct 2018 16:50:12 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Yuri Pankov , Ryan Stone Cc: freebsd-hackers@freebsd.org References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> <27f7681e-7870-ae21-0563-2f31e58d1603@yuripv.net> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <1fc28b9a-14ea-2e81-22f0-6118c7c2917c@FreeBSD.org> Date: Tue, 23 Oct 2018 16:50:11 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <27f7681e-7870-ae21-0563-2f31e58d1603@yuripv.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="osXjoQ2SazZLGVeqUB8gY19BAadNuaI3F" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 13:50:13 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --osXjoQ2SazZLGVeqUB8gY19BAadNuaI3F Content-Type: multipart/mixed; boundary="hICCJ5fv5FSaazg8nZDpJgGrQtn0m9F0d"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Yuri Pankov , Ryan Stone Cc: freebsd-hackers@freebsd.org Message-ID: <1fc28b9a-14ea-2e81-22f0-6118c7c2917c@FreeBSD.org> Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <5bbf4ebe-c62a-3c6d-617f-d1c832d87cf5@FreeBSD.org> <27f7681e-7870-ae21-0563-2f31e58d1603@yuripv.net> In-Reply-To: <27f7681e-7870-ae21-0563-2f31e58d1603@yuripv.net> --hICCJ5fv5FSaazg8nZDpJgGrQtn0m9F0d Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 23.10.2018 16:48, Yuri Pankov wrote: >>> Adding -fno-optimize-sibling-calls to the compiler flags will elimina= te the TCO. >> How could I add it for kernel properly?! Adding >> "CFLAGS=3D-fno-optimize-sibling-calls" to make.conf affects modules bu= ild, >> but not kernel itself (I'm surprised, to be honest). >=20 > Try using: >=20 > COPTFLAGS+=3D -fno-optimize-sibling-calls Yep, thnx I should better read documentation. --=20 // Lev Serebryakov --hICCJ5fv5FSaazg8nZDpJgGrQtn0m9F0d-- --osXjoQ2SazZLGVeqUB8gY19BAadNuaI3F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvPJxNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R49YmQ//TsOEjzF2Vlt+bLXXfJzBjJwN8hQ9tBLhcNdmdzuQu1p930uWxImifiVf hfamvsgsgehgw3SuSFhwMgS9vcT55M0p8jVA7+ZEFBES2WLhP0rwnnnzk7rjnQIZ qCPifsrIrZwv50y7am1v66t1N7JMBfm9mGUZndeTcixVYoDrSc8cZzBlcUyOpaF2 H0C4tqNuhhveADvJlqGBqdI4UDC04aL7xVSwmnDbP3nl4Ppo3yl3Ox5zBXY/4E08 Fw+yls6Lj8IPcAVrTTn2IJEYm3cOt3jnqhvciUXy6UC5sNJP5ddQsgPup4QnFdni z39iiJbTi108UHtrjYKhMzmsS6qVWqKnxZn15ipdTQLkoUjrI8xIAA3VyIX9me7o aSq3Dk/5WKOndKD2VsKbdcSLw4ajVSekJMzdDwjMC8gv9221t2yGgNBTNzY7yZTv VD9BVvNNhBDVWy/M23O73RVTKeKbfj2r3pNL56dsyu5T1297/9L2jdeGsvsZ7Otr eOPw4HGpdYNbQQwZXkQEvJcAlkXPavwC73Oklxd/FZ06qkUE6H2TSksJIgeoVXNj /en4adMKbgwtKrv7hVSNqXBu3X6Y3hQRmGCfzbi90/q/pA3UIarifJ8zCJAeNoO1 bJvvNPp1oPmXHC31MNaxX/VBRV8fetyHCE+hT4a8pMgPTjrE9xg= =xkS7 -----END PGP SIGNATURE----- --osXjoQ2SazZLGVeqUB8gY19BAadNuaI3F-- From owner-freebsd-hackers@freebsd.org Tue Oct 23 21:34:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8F3BFFD82F for ; Tue, 23 Oct 2018 21:34:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 40E02714A8; Tue, 23 Oct 2018 21:34:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [94.19.235.70]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id EFCEDBAF; Wed, 24 Oct 2018 00:34:11 +0300 (MSK) Date: Wed, 24 Oct 2018 00:34:12 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <168122586.20181024003412@serebryakov.spb.ru> To: Ryan Stone CC: freebsd-hackers@freebsd.org, cse.cem@gmail.com, Alan Somers Subject: Re: What is wrong with dtrace's stack()? In-Reply-To: References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 21:34:13 -0000 Hello Ryan, Monday, October 22, 2018, 11:50:29 PM, you wrote: > Adding -fno-optimize-sibling-calls to the compiler flags will eliminate the TCO. Stacks are better but still strange. For example I have such stack (which I like better than previous): kernel`lock_delay+0x72 kernel`sosend_generic+0x112c kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cebf6d According to addr2line `sosend_generic+0x112c' is https://svnweb.freebsd.org/base/head/sys/kern/uipc_socket.c?revision=339419&view=markup#l1579 Which is call to protocol-specific send function. Where is this function (it should be tcp for sure)?! -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-hackers@freebsd.org Tue Oct 23 22:08:48 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BDBBFFECD9 for ; Tue, 23 Oct 2018 22:08:48 +0000 (UTC) (envelope-from marius@alchemy.franken.de) Received: from alchemy.franken.de (alchemy.franken.de [194.94.249.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "alchemy.franken.de", Issuer "alchemy.franken.de" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 09AD872F1A; Tue, 23 Oct 2018 22:08:47 +0000 (UTC) (envelope-from marius@alchemy.franken.de) Received: from alchemy.franken.de (localhost [127.0.0.1]) by alchemy.franken.de (8.15.2/8.15.2/ALCHEMY.FRANKEN.DE) with ESMTPS id w9NM733l021066 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 24 Oct 2018 00:07:03 +0200 (CEST) (envelope-from marius@alchemy.franken.de) Received: (from marius@localhost) by alchemy.franken.de (8.15.2/8.15.2/Submit) id w9NM73MJ021065; Wed, 24 Oct 2018 00:07:03 +0200 (CEST) (envelope-from marius) Date: Wed, 24 Oct 2018 00:07:03 +0200 From: Marius Strobl To: "Amartya, Shreyank" Cc: "freebsd-hackers@freebsd.org" , "mav@FreeBSD.org" Subject: Re: FW: eMMC AMD platform Message-ID: <20181023220703.GF21523@alchemy.franken.de> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (alchemy.franken.de [0.0.0.0]); Wed, 24 Oct 2018 00:07:03 +0200 (CEST) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2018 22:08:48 -0000 On Tue, Oct 23, 2018 at 11:46:33AM +0000, Amartya, Shreyank wrote: > Hi, > > I have an AMD eMMC 5.0 controller connected to an eMMC device. > I recently got around to enabling HS400 mode but the performance is still same what I was getting in HS200 mode. > Is there a way to verify the HS400 mode is active? Hi, apart from examining the signals between SDHCI controller and eMMC chip with a scope, I don't see a way to verify that HS400 actually is active. However, if controller and device have been successfully switched (i. e. including tuning and none of the operations in the associated sequence has failed on either end) to operate at HS400 and data can be transferred between them without errors, there's no reason to believe that HS400 shouldn't be active. If both sides don't actually use the same transfer mode, transactions simply would fail. > > Patch for enabling HS400: https://reviews.freebsd.org/D17644 > > In order to enable the HS400 mode I had to clear the Sampling clock select bit (Host control 2 register) and then set it later (as part of the patch). > Now, I suspect that this led to reset of the tuning circuit and that even though HS400 mode is enabled, since it is not tuned, I do not see increase in performance. Could that be a possibility? Well, with a controller that has known bugs in that area speculating about what's really going on doesn't make much sense but generally what you describe isn't plausible; if an SDHCI controller isn't tuned to the device in a UHS-I mode requiring tuning, the result is - depending on controller implementation and operations attempted - an interrupt with the CRC and/or tuning error bit set in the SDHCI interrupt status. While I haven't seen a case so far where HS400 doesn't yield more throughput than HS200 (assuming that the eMMC chip actually can outperform HS200 in HS400 mode, which isn't necessarily the case depending on model and manufacturer and whether e. g. pSLC is used), one thing to keep in mind is that sdhci(4) supports SDMA at most so far, i. e. none of the ADMA modes. In turn, that AMD controller might perform badly with SDMA. Off-hand I'm not sure whether the alignment restrictions of that controller for both ADMA and SDMA might impact one of the modes more than others. In any case, you'd need to disable the use of the ADMA modes in Linux in order to get to throughputs that make sense to compare (there are other cases despite DMA modes where either FreeBSD or Linux takes better advantage of MMC and SDHCI features, though). Marius From owner-freebsd-hackers@freebsd.org Wed Oct 24 15:48:54 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FE2DFF1AA2 for ; Wed, 24 Oct 2018 15:48:54 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF4376A80B; Wed, 24 Oct 2018 15:48:53 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-lf1-x136.google.com with SMTP id n3-v6so4363199lfe.7; Wed, 24 Oct 2018 08:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TiUMubkGlefpV52oJ2EoKnW/OllSXGqnYX9pmmBDUAA=; b=EYDlh1KeMSwdSGI5GURFf6fc3844DIwFBoMpK1zlhQQNYvFvt/xHm1CPP4KsaXktgN GFZKyU4byzfKRVD81XnKnyYbuU4xb13JGCkngJaso1GliErNctdE38UwN+OyDuoOPi26 KkW6UWk7315j6MzrzZ/wGyCwaVNlPNesRVyYzWUg70647CeFifj6I1kjtJhKX6tDTtVN rHItqQPzbRuAp6W9W+Fm/jv/SmV6aU0CSf5ExVinp28IUpZHxmvNx5U0RZ3OiYwtALin 9LQdJLBi6Bol9FRcCBMoTYHiGFj+CT0cjTrZR+LOqmr3UhZDcXFDgnhVndsvvijCL2/T k3mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TiUMubkGlefpV52oJ2EoKnW/OllSXGqnYX9pmmBDUAA=; b=DICC3yTDIUoYnYUcFarhTj+LTbkse/4h4PyubKxn5oNGCcX9F+ITKqGHBRuo5/4c8t +lyQv71O3kzIx38w5GwfqJXfmnZneMX8Mq3Ofhlz+jgwOAxIUxNfXwNHHj7eQRYiLtUb wvWYhgNEwoVDTIzqT8pAhXXXHVfsJiBenIyIiDrcKmKI8H1LKmJ/1P9KGHEbIRnzWgOs m7+aTAHDS/1cksSx2XXd4gVM1ChuzT1hIFScz2fnxpjortyFY1xs7caewgWs8V0yJMIg td01+4j9nXtXcI+DTqsZueJtULMAky6k93BT5mXWuo+jEkoamm6Boydd3UPEpVFV+r7W svpg== X-Gm-Message-State: ABuFfogyXf5ZU4Z+GdtBAFuNj6DJM68nZNc0ljF9EqGKvpbJw+JsAztN VfAqu2DbTCTJESpyGxG5IYlG2tGSimCSlKH/3NVbDYUC X-Google-Smtp-Source: ACcGV63pkb+MEhwUR22j1jyPcyB+VEbsfXSp9ejohZES8HZcLOmDu2gZP/syN73I0wt8Hoh6XQisYwTUlKvkB5Q8jJA= X-Received: by 2002:ac2:434d:: with SMTP id o13-v6mr15910372lfl.129.1540396132003; Wed, 24 Oct 2018 08:48:52 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> In-Reply-To: <168122586.20181024003412@serebryakov.spb.ru> From: Ryan Stone Date: Wed, 24 Oct 2018 11:48:40 -0400 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: lev@freebsd.org Cc: freebsd-hackers@freebsd.org, Conrad Meyer , Alan Somers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 15:48:54 -0000 sosend_generic+0x112c is the return address, so it's one instruction after the call instruction that called lock_delay. What's the line number of sosend_generic+0x112b? On Tue, Oct 23, 2018 at 5:34 PM Lev Serebryakov wrote: > > Hello Ryan, > > Monday, October 22, 2018, 11:50:29 PM, you wrote: > > > Adding -fno-optimize-sibling-calls to the compiler flags will eliminate the TCO. > Stacks are better but still strange. For example I have such stack (which I > like better than previous): > > kernel`lock_delay+0x72 > kernel`sosend_generic+0x112c > kernel`sosend+0x79 > kernel`soo_write+0x6b > kernel`fo_write+0x4a > kernel`dofilewrite+0xcd > kernel`kern_writev+0x79 > kernel`sys_write+0x8f > kernel`syscallenter+0x774 > kernel`amd64_syscall+0x1b > kernel`0xffffffff80cebf6d > > According to addr2line `sosend_generic+0x112c' is > > https://svnweb.freebsd.org/base/head/sys/kern/uipc_socket.c?revision=339419&view=markup#l1579 > > Which is call to protocol-specific send function. Where is this function > (it should be tcp for sure)?! > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > From owner-freebsd-hackers@freebsd.org Wed Oct 24 16:10:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55A40FFA9EC for ; Wed, 24 Oct 2018 16:10:49 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id CE2E26B9DA; Wed, 24 Oct 2018 16:10:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 985A3DF6; Wed, 24 Oct 2018 19:10:47 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Alan Somers , Conrad Meyer References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Wed, 24 Oct 2018 19:10:41 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 16:10:49 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6 Content-Type: multipart/mixed; boundary="12uJTI8sAMiThSkLuym1l9QItCGS2mjQC"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Alan Somers , Conrad Meyer Message-ID: Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> In-Reply-To: --12uJTI8sAMiThSkLuym1l9QItCGS2mjQC Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 24.10.2018 18:48, Ryan Stone wrote: > sosend_generic+0x112c is the return address, so it's one instruction > after the call instruction that called lock_delay. Yes, but this instruction (+0x112b) calls real protocol send function via function pointer. > What's the line number of sosend_generic+0x112b? It is here, uipc_socket.c:1589: error =3D (*so->so_proto->pr_usrreqs->pru_send)(so, (flags & MSG_OOB) ? PRUS_OOB : /* * If the user set MSG_EOF, the protocol understands * this flag and nothing left to send then use * PRU_SEND_EOF instead of PRU_SEND. */ ((flags & MSG_EOF) && (so->so_proto->pr_flags & PR_IMPLOPCL) && (resid <=3D 0)) ? PRUS_EOF : /* If there is more to send set PRUS_MORETOCOME. */ (flags & MSG_MORETOCOME) || (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0, top, addr, control, td); I have these three DIFFERENT stack, which should be one for sure, as it is confirmed by custom SDT probes now (I've added them and checked). Addresses are different from previous examples due to some code was shifted by SDT probes. 26462 times stack is Ok, but 4068+3993 times some frames are lost. Same address could not call both tcp_usr_send() and ia32_pause() and lock_delay(). Really, it should be one stack trace! kernel`lock_delay+0x72 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 3993 kernel`ia32_pause+0x7 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 4068 kernel`ia32_pause+0x6 kernel`tcp_usr_send+0x131 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 26462 > On Tue, Oct 23, 2018 at 5:34 PM Lev Serebryakov wrote= : >> >> Hello Ryan, >> >> Monday, October 22, 2018, 11:50:29 PM, you wrote: >> >>> Adding -fno-optimize-sibling-calls to the compiler flags will elimina= te the TCO. >> Stacks are better but still strange. For example I have such stack (w= hich I >> like better than previous): >> >> kernel`lock_delay+0x72 >> kernel`sosend_generic+0x112c >> kernel`sosend+0x79 >> kernel`soo_write+0x6b >> kernel`fo_write+0x4a >> kernel`dofilewrite+0xcd >> kernel`kern_writev+0x79 >> kernel`sys_write+0x8f >> kernel`syscallenter+0x774 >> kernel`amd64_syscall+0x1b >> kernel`0xffffffff80cebf6d >> >> According to addr2line `sosend_generic+0x112c' is >> >> https://svnweb.freebsd.org/base/head/sys/kern/uipc_socket.c?revision=3D= 339419&view=3Dmarkup#l1579 >> >> Which is call to protocol-specific send function. Where is this funct= ion >> (it should be tcp for sure)?! >> >> -- >> Best regards, >> Lev mailto:lev@FreeBSD.org >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 --=20 // Lev Serebryakov --12uJTI8sAMiThSkLuym1l9QItCGS2mjQC-- --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvQmYFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R48MaxAAyiUF7zIrHfmfyvijJJKVBzZk8dL+ko0sUuqonaHaP/BLk088KAabpKd+ 4nqnolCcpRlmenenR54t6EvDKXAkhKvH4Qy52NAjU699uzqBDcnk4FKPpq2gKZbE Dz7WUb8itARTeq7ztFoJ00VPsLZSFoAWipuhpwvDwM1U/J54Iprh7v3rYlW7LJNJ 88Cd/xsxjt9ZtNw/F6YbncsgAIrDxSSTdAUa2oztcHoyM5x7PqrsoXcjQ8Ln2Pir ZNUFd8vI62z44IVHnuNRX/1mmWRBHFRkgnkfTKEimJIcK9ZfAccP4bS6OEJBYLLA 0c/xougcb7JJ58EWLXWQk1qzVxozGapWLozpWADSgc5AYi1g3xYlyA8Z9Qyy0emF InmKWnGHe7JPYPd5VnyI8hABH3PbQKuelmfEKgytjDrhxKXL+EWupgw20MLJhZcG 2VKEvaenoYVGYUUhhPvZ10PMQ69ulmWelC2J4n1xOYNdFLa/M3/98re8RhySuci2 bd7uk/7UTTDFIl7/y1r06Nnwjqe6ozo6ooiIHd6U0y/54iXpHO+rEKkBikTrPFF6 tHSt+8rMoP0OKq/V2U7QTulLFDPnEaanFS1Z06tw1wAVnqL3vGHdoFpcxPY4MoYZ a9w59fSX7iS4LAeUdlrwM5FMCGUx3gr5pYVJznzVEjOjOebrRTU= =KiLK -----END PGP SIGNATURE----- --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6-- From owner-freebsd-hackers@freebsd.org Wed Oct 24 17:14:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10CBCFFD79C for ; Wed, 24 Oct 2018 17:14:24 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 559A96F1E8; Wed, 24 Oct 2018 17:14:23 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-lf1-x129.google.com with SMTP id w16-v6so2459042lfc.0; Wed, 24 Oct 2018 10:14:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Xx1IL9Pl5MhSW9j6p/D+ZKNHJtwTT0mcxrmj0ww9vLw=; b=V3DRpySNFWN83ZYFusBLNEmZuPOMuHNUf0nwqGugiKWYXtJzXD0sb8xKA1RtHylB+E vFQXl8hAkmGZJQakaS4Tayfpu06qI2og276k9lazYKwgBZHBVL1ECjtkCMFGjOfGWLgz WR4+9mHHEcelRwTJ0x0AY6kPbeQfufJpC2hchlaX3JoJ8MhXe0tDCRrVJOEBcvIS3z1g eva7Tfh5h5GmuRu4hEyEBLWVnY9AxXV/vInRXQ6KuGJ00mdDUK26ur4RmN9hNPZZnRr1 OjW5xXIhUVX+ioDxN2DxOUmfJCEISg3VPKO4Nm1kVO4u0Or9CfwuBDdHJDMXAoWJ702A XZaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Xx1IL9Pl5MhSW9j6p/D+ZKNHJtwTT0mcxrmj0ww9vLw=; b=CI0Gw8DdiIQIhUxtG6PFW7Kfg6arTcxVN1Zv+7uHsndrDrj9xkatvKC4ypUnXnvfGU h+Y2SO5qYxxqeOs0qFED/hiIVP78oW5gBTvifK/F7o6SxdL3+PviZGZHQmgY1LKL7AlY iv5NV79cdHankwYHHkZEWU2eHuhe8Wqj+5DvbfMiufVqZwBQLooSI8Hfn8VmZVwhhnNd m+lRX2sCo3OJtl3LAWdsQfX4oW+WWvk/B0KEvse0UzLGOu4Rk/w357FwNfMKiSc97u7n pkLd+OBo7JR6QeITvb9M9jhs3ZdcO9C/H7+tkjoPgrbbHhf3YUI6l5ZH6Pf6PiobYcuJ 5BmA== X-Gm-Message-State: ABuFfogUags1RTyAFaz5dQS3VtrbImhNyLbQkRAUMPvzWEzGtNGBIZ7X uEHxn+FHw6p1CNr9AZXyBlPRypaP0xfYPPdBsTE5hA== X-Google-Smtp-Source: ACcGV62/Qpl6wmA7E81k0eTkhphLTSLoFaKqRsNMC3ibecFZDoPDtMjGlkM8+5rBHYL5bGYKd+DGefS/BGXSeq42SzQ= X-Received: by 2002:a19:8fce:: with SMTP id s75mr15125447lfk.151.1540401261657; Wed, 24 Oct 2018 10:14:21 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> In-Reply-To: From: Ryan Stone Date: Wed, 24 Oct 2018 13:14:10 -0400 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: lev@freebsd.org Cc: freebsd-hackers@freebsd.org, Alan Somers , Conrad Meyer Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 17:14:24 -0000 ia32_pause() is an inline function. How does dtrace map instruction pointers to symbol names? Is it getting that mapping from some CTF data, and is that CTF data aware of inline functions? If so, I'd argue that behaviour is counter-intuitive and unhelpful, as this example here shows. From owner-freebsd-hackers@freebsd.org Wed Oct 24 17:19:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7192AFFDA15 for ; Wed, 24 Oct 2018 17:19:25 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9A036F4B2; Wed, 24 Oct 2018 17:19:24 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pg1-x532.google.com with SMTP id g12-v6so2647626pgs.1; Wed, 24 Oct 2018 10:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=uSIZHFG8hcuWuTMTiaQmi5GUWmlMmXvgqDoOWrUZLNk=; b=lokqwz8EUEiArQMc6Ww5Btrq3OG9OTjYXPul8tsamaGdmPKIS8Cp+p07a8YyEVYqeI lyjNRmyLmNNV8ck+0gyy6TMl6Uh4kCtRnLHGQ+J1aQ+9GptdVJG+Xr6F9z0nJNLJTvs4 t0faLspr9/gZOoNnfZfXwdLlNQaqCclsbpsuwW7fQ+bUl6A57vOw/DjMNoFZ9/wPvLHD 9/AvjpOFN17Q3wK5SzaJZ8y0XZA7sCQAUDQc8vFSpVeE9Ljq4AYs/7xXlfWVRazU6ano 7XWQ35fkdd3IXbkKmZKUbDCgi+ucZWHdGPIMQo/R09Fg7zv7exmRwAG8nMrPNbEQb9ct T1DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=uSIZHFG8hcuWuTMTiaQmi5GUWmlMmXvgqDoOWrUZLNk=; b=qSBmu9r/oDECpyQM5R9PsdT50QEuYAmaXdZXWQcf+spyxu63T10PtqI2acgpkkbLVA b90pGF1IgiOmvMwmX6gEBid76eDRGzkE6+kBi/4uNQdRIcujpSW5mnHcU6glGDub23sO AiQnxdHZuDlm6ypl26nusB0qI96Wc0OGs3/dIpfHuDCvFyXaW6YURLgZjhpFhRGWGpYJ lg8UxZS/UNiwNNk6X3aFl0WxA7ICRes633EMVaS4w+CV8xpOyLfJ2s4TwkAZL9iTe7Rx IbbAjiVzFNoXbCJrGcbscYtUw3nUVQ2WEB2CgRWLa8O2XPoUG3M5XrHxp/bReEcEH0tD vnWA== X-Gm-Message-State: AGRZ1gLoUlXTrzPz/5Jld5IhtzS5PlmiLr2bryC/ip8u/TehNv4hFDaj DNXPmxa3nBWCbwcRMXpo0c09x10= X-Google-Smtp-Source: AJdET5dkOlp/hF/2l5XHvvztoRVD4C4ACSTeqCVcGS9tpjbpoPjVDdPE0PGXgGqWDLhtcNalzrxLtg== X-Received: by 2002:a62:418a:: with SMTP id g10-v6mr1005449pfd.44.1540401563129; Wed, 24 Oct 2018 10:19:23 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id r23-v6sm7007328pfh.79.2018.10.24.10.19.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 10:19:21 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Johnston Cc: freebsd-hackers@freebsd.org References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> From: Robert Message-ID: Date: Wed, 24 Oct 2018 10:19:20 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <20180928152550.GA3609@raichu> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 17:19:26 -0000 So the issue is still happening. Please check attached screenshot. The green area is "inactive + cached + free". At some point massive amount of "Free" memory is moving to Swap and Laundry. Then slowly returns back to inactive. My concern is that I've tried all kind of monitoring for mem allocations, including dtrace-ing of all mmap calls. It shows no any allocations of such a huge size, so I believe this is something related to the kernel's mem management. Could you advice which other places should be checked\monitored to find a root cause? Thanks. On 9/28/18 6:25 AM, Mark Johnston wrote: > On Thu, Sep 27, 2018 at 04:04:15PM -0700, Robert wrote: >> Is there a way to force move pages back from laundry to Free or Inactive? > There's no real way to do that from outside of the application. The > application itself can free anonymous memory by unmapping it. It can > also force memory to be marked clean (thus moving it back to the > inactive queue) using madvise(MADV_FREE). This will cause the memory's > contents to be discarded, though. > >> Also, what's the best way to identify addresses of these pages and >> "look" inside of them? > There's no convenient way to do that that I'm aware of. On amd64, you > could in principle use kgdb to dump individual pages in the queue via > the direct map: > > # kgdb82 -q > Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. > done. > sched_switch (td=0xfffff8002c560000, newtd=0xfffff800025ca000, flags=) at /home/mark/src/freebsd-dev/sys/kern/sched_ule.c:2112 > 2112 cpuid = PCPU_GET(cpuid); > (kgdb) p vm_dom[0].vmd_pagequeues[2] > $1 = { > pq_mutex = { > lock_object = { > lo_name = 0xffffffff80c34c0d "vm laundry pagequeue", > lo_flags = 21168128, > lo_data = 0, > lo_witness = 0xfffff8041ed6cb00 > }, > mtx_lock = 0 > }, > pq_pl = { > tqh_first = 0xfffff8040f9ef980, > tqh_last = 0xfffff8041227f448 > }, > pq_cnt = 20087, > pq_name = 0xffffffff80c34c0d "vm laundry pagequeue", > pq_pdpages = 0 > } > (kgdb) x/512gx vm_dom[0].vmd_pagequeues[2].pq_pl.tqh_first->phys_addr + (vm_offset_t)&dmapbase > 0xfffff801aea00000: 0x0000000800000000 0x000000082298c400 > 0xfffff801aea00010: 0x00000009be801000 0x0000000000000006 > ... From owner-freebsd-hackers@freebsd.org Wed Oct 24 17:23:07 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBA11FFDE46 for ; Wed, 24 Oct 2018 17:23:06 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x144.google.com (mail-it1-x144.google.com [IPv6:2607:f8b0:4864:20::144]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66FFC6FB3E; Wed, 24 Oct 2018 17:23:06 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x144.google.com with SMTP id m15so7596804itl.4; Wed, 24 Oct 2018 10:23:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=3Ox9HWNiZkEaQYY9+ILgJUrPajvUOAJCp15D0R7DRvo=; b=bQZmDL3PV+9W8x+wQpas6IqzJRiIszr+pEW5JJ2Ob2BfdL2mTsT0uYxNqr//CMQVYN nrChQzu30xpwya3vdd6HLWxCHnwF6sktBmj7thXF3UkfGkOn3xkOhkgef1/LX3PyOKb3 rcHfumo8nXQh4BsWYcdEJU7S5HSOnXitJpC7cXo9yYHqDQFngsCtoMclBe2jg37DPYdG etPzFGlpo2UjP2SgOlSVb98wdK++0cmSZPo/Redg7C5jydu5KqImler12T2eYezNSojq gkP3wPIWBTXFpugy9z7yZLO2NB+l6riQQ5MbUxQFHnEvCDV07mpIOje3Sm8vBy5GclIz tgEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=3Ox9HWNiZkEaQYY9+ILgJUrPajvUOAJCp15D0R7DRvo=; b=TlCXh3907OQ46NwbOxXloenLl/g8X+//ap5faVPonsAzODtk7fz+xz18S7e60oTwfQ LwF0irNjVB133gov0FaPDss8TeE7JHdewxtFCjxuTA4xn3PMaD5Cmhy1tr4gEcOB2nmz moCvGZVKlAaeW3oLI5LXmPjCnO58JSWCj0GczRMvaDHWnuI+7Vb/QR3IKiz+ilnk0jDm BGvUDuhUBWk5nEQu0SPGgsOepfDTNZQts3Lwp40d6GGI+v0SVHdd9et/Mb486OSe+7J1 xTzk+TfSnjquDjwQlOGops2CxM4ck2vJM/rQEWfFwDrw1N536wPrHoUZVFBy0XP1ALPc DDhQ== X-Gm-Message-State: AGRZ1gKBI036+55Rd4b7eAtgtzvcTXAJ0PunLGITtVddk8AIvW1XM5+y u6s+ZFqf/PQPMmA16gXlbYs= X-Google-Smtp-Source: AJdET5cXJIMu48EikNYUflQuxoNkWq6xp15gvUEWEr+9SnljBe9bKil3U7sOxHSG0Yk0mdV9UKfH8A== X-Received: by 2002:a02:b5c1:: with SMTP id y1-v6mr2496296jaj.143.1540401785517; Wed, 24 Oct 2018 10:23:05 -0700 (PDT) Received: from raichu (toroon0560w-lp130-08-67-71-176-199.dsl.bell.ca. [67.71.176.199]) by smtp.gmail.com with ESMTPSA id t25-v6sm2696093iob.73.2018.10.24.10.23.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 10:23:04 -0700 (PDT) Sender: Mark Johnston Date: Wed, 24 Oct 2018 13:23:02 -0400 From: Mark Johnston To: Ryan Stone Cc: lev@freebsd.org, freebsd-hackers@freebsd.org, Conrad Meyer , Alan Somers Subject: Re: What is wrong with dtrace's stack()? Message-ID: <20181024172302.GF45118@raichu> References: <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 17:23:07 -0000 On Wed, Oct 24, 2018 at 01:14:10PM -0400, Ryan Stone wrote: > ia32_pause() is an inline function. How does dtrace map instruction > pointers to symbol names? It uses a kernel linker API whose implementation searches the symbol tables of the kernel and loaded KLDs. > Is it getting that mapping from some CTF > data, and is that CTF data aware of inline functions? No, and no. CTF does include a function table, but that's just for encoding function parameter and return types. Functions that get inlined into every caller (as I'd expect with ia32_pause()) shouldn't be showing up in stack traces. There is no ia32_pause() symbol in my kernel; presumably the observed behaviour is related to some non-default compiler flags being used, but I haven't carefully read through the whole thread to see. > If so, I'd argue that behaviour is counter-intuitive and unhelpful, as > this example here shows. From owner-freebsd-hackers@freebsd.org Wed Oct 24 17:26:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBF28FFE092 for ; Wed, 24 Oct 2018 17:26:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 4865E6FE8C; Wed, 24 Oct 2018 17:26:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id E3748E35; Wed, 24 Oct 2018 20:26:33 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Conrad Meyer , Alan Somers References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Wed, 24 Oct 2018 20:26:21 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7BrRdxDYahHwCnPlh3S6ZYNNhCexDf7a6" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 17:26:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --7BrRdxDYahHwCnPlh3S6ZYNNhCexDf7a6 Content-Type: multipart/mixed; boundary="U3JEUfUuyBAemrdoU3tZDIQLpBBTsohOn"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Conrad Meyer , Alan Somers Message-ID: Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> In-Reply-To: --U3JEUfUuyBAemrdoU3tZDIQLpBBTsohOn Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 24.10.2018 20:14, Ryan Stone wrote: > ia32_pause() is an inline function. How does dtrace map instruction > pointers to symbol names? Is it getting that mapping from some CTF > data, and is that CTF data aware of inline functions? It looks like plausable explanation, but why sometimes it pick-up "middle layer" like this: kernel`ia32_pause+0x6 kernel`tcp_usr_send+0x131 kernel`sosend_generic+0xf61 And sometimes doesn't: kernel`ia32_pause+0x7 kernel`sosend_generic+0xf61 As I said, these stacks are equivalent, I've proved it (ok, it is not mathematical proof, of course) with custom SDT probes around this "`tcp_usr_send+0x131" place. --=20 // Lev Serebryakov --U3JEUfUuyBAemrdoU3tZDIQLpBBTsohOn-- --7BrRdxDYahHwCnPlh3S6ZYNNhCexDf7a6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvQqz1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+sbBAAoNs2RjYO8mnrJbqfH18OsvJ7U/dwnxnlaC/gYWZ6BAdfvQV0P0mDFNxS 2lD4K8k7SWQTJIesaVwqk13kTg1ICvBs/WxUlSSEZTeFUwnBxkQRYlqPT6hnvaeX olXt/S0PPrpTY0vMywP7OnkAbWziYq0r3d3lnKkj15kgcMkAPNubK2JdlF4/J1Fg Sz8kwJ5BjMbemiDGL5PpX7bYfthsFpFkjvlrius6hhEiSvdvQ1l6TED5gGqP6660 qZ8DG0/NUeX42WIlMhgYVRBEkNooALlUqdq+LVdT/tM/x2VI3pfICgWt6iy/sYgl 5dG55w8DW4kTfzU9yXpcGshM3sSKToykzKbbF6yoQgoBz+J9i/Z4N+mOfb0Brfr7 bs3QKs34oZs0hv76VbIzujClnlQScW18sWvAudfFwV1dUPMa1GZkzEqZt4CYO4OQ 0Vpz/BbhrZpdrz7qp1WMaau+5UJ6bo8wFxJSgzZJoXzSMZ8PFmntP8zMVKsnQVLg HylYELI3iD3nCSWba4nFE18735J2lLe9f0qCxTqIkHCZqdTQKvzY0gXoZ3v/UKmp Nwnx5d4w2U66mXER8cTfeX8gp5k4IJernXbTae6A2Nhgj94oKfEs2D/vg+l0lK1+ Ekd8X1ICe/lrfE9niBFxDXwHIfq4SeePZB/ktnxg0LUjaGGw1kE= =kTcY -----END PGP SIGNATURE----- --7BrRdxDYahHwCnPlh3S6ZYNNhCexDf7a6-- From owner-freebsd-hackers@freebsd.org Wed Oct 24 18:12:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5F07100803F for ; Wed, 24 Oct 2018 18:12:41 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FBF373026; Wed, 24 Oct 2018 18:12:41 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: by mail-ed1-x531.google.com with SMTP id r25-v6so5968591edy.0; Wed, 24 Oct 2018 11:12:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:date:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/WqB9SL2yixHVeOKRElE0I9+wppfgr87RGiaHnGpNxU=; b=iWQ2Nta/nT9EKw26L9erzlKqf6iYlce31oTSgc97zk8QqZeNGx/L+DIV+UNvMpQexm IlUjyeLsCxx6Yq+5gJ6e5cqKIW1N6UXdGLkPWCE1tEPGt2Zm2SxA8zqIPeJfEoyoE/5Z 8D2SXDvr2l08ihlx18ebcsZDdZ84fa4bogoO2G/3QcoFvjS/70nPQy2A1zpoVd3+jHYZ uF3UodvsIFbmXgalMc5Hg+ToGpQSrxXI41p3+O6FAT65XQlcPJDsnFXUvf7/r30fk+ZC abL5zxycGxIoEdV1+ps41dNTkngR7c7tPwDJ9hynqYVzoOyY1j0i2ftFxNfaD+/15Pez xRvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/WqB9SL2yixHVeOKRElE0I9+wppfgr87RGiaHnGpNxU=; b=rQHWR0RlcHvtK8O02BivS3kWozBAOQgjj79SL6eO5Er1o4mNodsF6fYe42ISkWK2Og fCHnaZc7I9vHFlfTyHPih4aswGw8fqslWxTIAcMQRfTnGw4W7Z3hL9HiReDez1BVH0PL 5YLPohizpamT8no0f2aGqP2Vh7tSGjAnNLXuXGwGUWK4TbaxiK5PuA+xrOVa1/Z/QsUO QyMLxHntq6ySQxegGaKIyKBmAf2BsfMIIAyl9m6CQ3bB3Ubw6jZCez/JRs0B9Zol1FuL t0ZnNnC1VlsvewHRNDqPPqhwDWTAmt6oNPwlx1ZExSmawHnJhIfO4RUz/wXxlwJ9aSb5 9wdQ== X-Gm-Message-State: ABuFfoiBwrX8eRmFMd175ARlIhNeGm60ulmPWAajBBQczdh0wFoCsHPR SZ0XlDhdK30K8D0hAHnHPaY= X-Google-Smtp-Source: ACcGV62nHpc34N1bJwRKpF/R62dmchA7RFjflHr3PYVmvo08if/HaU6yJYcpsqWby7XJnC+w46WFGw== X-Received: by 2002:a50:d303:: with SMTP id g3-v6mr19781122edh.206.1540404759919; Wed, 24 Oct 2018 11:12:39 -0700 (PDT) Received: from localhost ([2001:470:1f15:3d8:7285:c2ff:fe37:5722]) by smtp.gmail.com with ESMTPSA id y47-v6sm2673544edc.27.2018.10.24.11.12.38 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Oct 2018 11:12:38 -0700 (PDT) From: Rozhuk Ivan X-Google-Original-From: Rozhuk Ivan Date: Wed, 24 Oct 2018 21:12:37 +0300 To: Robert Cc: Mark Johnston , freebsd-hackers@freebsd.org Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <20181024211237.302b72d9@gmail.com> In-Reply-To: References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 18:12:42 -0000 On Wed, 24 Oct 2018 10:19:20 -0700 Robert wrote: > So the issue is still happening. Please check attached screenshot. > The green area is "inactive + cached + free". > > At some point massive amount of "Free" memory is moving to Swap and > Laundry. Then slowly returns back to inactive. > > My concern is that I've tried all kind of monitoring for mem > allocations, including dtrace-ing of all mmap calls. > > It shows no any allocations of such a huge size, so I believe this is > something related to the kernel's mem management. > +1 Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free Swap: 112G Total, 19M Used, 112G Free Looks very ugly from user side. From owner-freebsd-hackers@freebsd.org Wed Oct 24 18:35:33 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8D0F103601F for ; Wed, 24 Oct 2018 18:35:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-3.consmr.mail.bf2.yahoo.com (sonic306-3.consmr.mail.bf2.yahoo.com [74.6.132.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60B037433A for ; Wed, 24 Oct 2018 18:35:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: WRVCOzAVM1lyxpw.ifC1ps.vVaoGzSTzPbGf1U_opJQw1OgGE9c73b9VnhKSsI_ iZ8oJWftl.cqDnLX6TJWrtv6ijycxyz5J0CW_fXLudy75baEBEbrK7Lyrc3CIuqm4YiweWesrymH wmP1o7eHQgf.euQg1pEeoN8Ztk2L4J7oTYKwsN0HzGtGU4Al.g1iXe9rr.saJwgBrdO7fcPcB.oZ oEGI1kA_MdBpEEZYonpR.SXGftKJdGMKAM0Ij.hQDbXVGd.QudmV5PYEsE1FWGtBlLXVA9EqalVl JeIsg6GdmFZSP_qNiReZueS3rk3k91LsCb0awEQVAnkHxl0gWsSQm7fnw4wmWBqoQ4AntjdKZSSC 4R.Avy70v_0ctCIoK9wToLsQc1yyVUxPLkDcH.nTGqZHgdXPO1N9QvahEjUbotNeAByt241RMngn RfCtsVppd9.Wu0IDYj0tLEoq5xhIXaERcWG2JRlUcqEWkN9DXPOUSYZmPBmGjyGtb5xdw_qxHQv1 hTFcHLiaKZkJNYToU6UAfmU.JJa5a0Iq3qYRtEFmRGV_hymXz65Ui6z44LZdDgE1OFYnLyCi6GD6 QLf0GBULEiuUMod7EbJDAz1l9o9rG9GAMG3mL09fpPF_SDs6SevNpcHmGKPClStLuCxCHf3DOr58 fczPVDEpYb1vgRZzggAVm87z1k_xGVYUKtQwo2KEN86sXadaAQhogFzUHUT7wvOjoF33_UHR2hMk xOpU_hRDAa1vSMgGzdzL3TAofhszjnISNfJKlwLnR91h1riSvKqawO20aUXT8wSfn.5InzIyt02b GiNH8qmKilpm9wYyGKiqlAZSYdps3UsEFj4Z_93hfppBSFlcOoHAqMUJIFb4OwXvfnFnSigIKBPf olXdWio2It3jZNpRHSTHgNOn5WCvDNkthUr8D6hHx7lmmYf5K5VeZNGwJD.eArRl1YWK_g5kCzpR waDDunIBg5yQgtGWRGFCKv0rUoTYpwOoULK7a5Y_pDyWBZOjhKtMZTUNHtcPoaiYwQ20Rm4rKBJ_ DqKBysjABp85_eQadLqbndHA0amgmk78jBLNwMTy1gE_C9yyt0kOzczooag-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.bf2.yahoo.com with HTTP; Wed, 24 Oct 2018 18:35:32 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp415.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 35478f8b76af0edb5c16d5260be74c32; Wed, 24 Oct 2018 18:35:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Sudden grow of memory in "Laundry" state From: Mark Millard In-Reply-To: <20181024211237.302b72d9@gmail.com> Date: Wed, 24 Oct 2018 11:34:17 -0700 Cc: FreeBSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> To: Rozhuk Ivan , Robert X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 18:35:34 -0000 On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: > On Wed, 24 Oct 2018 10:19:20 -0700 > Robert wrote: >=20 >> So the issue is still happening. Please check attached screenshot. >> The green area is "inactive + cached + free". >>=20 >> . . . >=20 > +1 > Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, = 617M Free > Swap: 112G Total, 19M Used, 112G Free Just a limited point based on my understanding of "Buf" in top's display . . . If "cached" means "Buf" in top's output, my understanding of Buf is that it is not a distinct memory area. Instead it totals the buffer space that is spread across multiple states: Active, Inactive, Laundry, and possibly Wired(?). In other words: TotalMemory =3D Active+Inact+Laundry+Wired+Free. If Buf is added to that then there is double counting of everything included in Buf and the total will be larger than the TotalMemory. Also Inact+Buf+Free may double count some of the Inact space, the space that happens to be inactive buffer space. I may be wrong, but that is my understanding. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Wed Oct 24 19:56:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5718110393CB for ; Wed, 24 Oct 2018 19:56:32 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CDF8877647 for ; Wed, 24 Oct 2018 19:56:31 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x141.google.com with SMTP id 74-v6so7759032itw.1 for ; Wed, 24 Oct 2018 12:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=3CySXw71emc+T3Vde8LkA9qpWimYblU/70jQIodFrIk=; b=KOM74fDhSSGY+fg8F5m3qNPFjZhqtMSzWr9BhrBttr3QPQ0OzUA5Uj9MjYcfnrGUol tLTz0FJ1ypVMGxn9Io75XTPqBFjToEAgl85PjtAYctlJwoXr8mEM8ErPB52j5J3FqF6J 97GATxfjarfLrzlyuUf45YBUCLf7zBQGP7hV547blG2YK1/2Lm2kwQnW9bbEsF9D1f/e lGkdCI5Twoztr1Apzhr+/RAHYZ1nmgnewjS8ah3v/jwJcCceN93U0qc0AuVaa8bHquqP xW7KrXc3dzLKQZJxoHSsXSUuXdjM48oOQlZBmVQACm01Rvs1TtlvXihtHsbvne6+KmZA EqFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=3CySXw71emc+T3Vde8LkA9qpWimYblU/70jQIodFrIk=; b=iCbtL8tucB6lx1eQPWj4VpUTJZEP58lkXLwm4RlMyemf8bW6tgwGXEPTAPV0DgGkTl iw1nlyUI46BqXJYiW0z+zzpfIZbX+C/rtquKR38xlys8Q65q4Z04AX5XlDV//AI9Wu9f DzYN+HyAk9LLYsednsJDI8jWPvGAFxBng8gkAqUYV6GlJ+wR4dqt+u9nY7EfpRnIFLmE P4Bc0V6v3d4aLNFaxKX/gU5u6cDu17To9dgCODlrovr/EFDsm5Ca9fOmNBHzvWqM4TeD oxInG9kB/wdQCaQ9sxQEU7PYGV0LvPg7Ec+H9Vn9gNLD62IZ+oiHQwNaDAjX72PbJ0oy mOZg== X-Gm-Message-State: AGRZ1gJyQ7F96ktdU8M+WCEialq6fXZ3EJ08aZ6tjFy/J+M1yVtRt7Pf 4VsMU8WT5GJef7uDp05Lb9R5/f3JoAk= X-Google-Smtp-Source: AJdET5eeJFJbc+tOGuokBpB8geoIkPNH9H1VIssReVWrMI/JU6R8FkoBkTZsKAPylCYm4++Ilq4BlA== X-Received: by 2002:a24:6907:: with SMTP id e7-v6mr2552190itc.113.1540410991019; Wed, 24 Oct 2018 12:56:31 -0700 (PDT) Received: from raichu (toroon0560w-lp130-08-67-71-176-199.dsl.bell.ca. [67.71.176.199]) by smtp.gmail.com with ESMTPSA id i17-v6sm2117673iog.56.2018.10.24.12.56.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 12:56:30 -0700 (PDT) Sender: Mark Johnston Date: Wed, 24 Oct 2018 15:56:27 -0400 From: Mark Johnston To: "Bjoern A. Zeeb" Cc: freebsd-hackers@freebsd.org Subject: Re: [CFT] capsicum patches for rtsol(8) and rtsold(8) Message-ID: <20181024195627.GI45118@raichu> References: <20181015194212.GA2751@spy> <20181016165308.GB5066@raichu> <86D87437-BD34-489A-87B7-33F1089080EE@lists.zabbadoz.net> <20181016200414.GD5066@raichu> <2A564C8A-FB64-4D2A-9E3E-392F1FCA66BD@lists.zabbadoz.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2A564C8A-FB64-4D2A-9E3E-392F1FCA66BD@lists.zabbadoz.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 19:56:32 -0000 On Mon, Oct 22, 2018 at 11:57:44AM +0000, Bjoern A. Zeeb wrote: > On 16 Oct 2018, at 20:04, Mark Johnston wrote: > > > On Tue, Oct 16, 2018 at 06:29:49PM +0000, Bjoern A. Zeeb wrote: > >> On 16 Oct 2018, at 16:53, Mark Johnston wrote: > >> > >>> On Tue, Oct 16, 2018 at 04:06:43PM +0000, Bjoern A. Zeeb wrote: > >>>> On 15 Oct 2018, at 19:42, Mark Johnston wrote: > >>>> > >>>>> https://people.freebsd.org/~markj/patches/rtsold_capsicum.diff > >>>> > >>>> (0) the git rename doesn’t really work when applying the diff > >>>> with > >>>> FreeBSD’s patch so the mv has to be done manually > >>>> > >>>> (1) the rtsol Makefile also needs cap_syslog and util to link to > >>>> otherwise rtsold.c has unresolved symbols > >>>> > >>>> (2) rtsol seem to have worked when manually invoked; > >>>> /etc/resolv.conf > >>>> was created (I had rm’ed it) and the 3 nameserver lines > >>>> re-appeared; > >>>> sorry can’t test the search string here > >>>> > >>>> (3) rtsold crashes: > >>> > >>> Thanks. I made some last-minute changes and forgot to retest, of > >>> course. :( > >>> > >>> I uploaded a new patch which should fix all of these issues - could > >>> you > >>> give it a try? > >> > >> With the old and new patch: > >> > >> root@i386-a3-carp:/usr/src/sbin/rtsol # rtsol vtnet0 > >> failed to run script: Invalid argument > >> > >> Hadn’t noticed that before. > > > > That's a cosmetic bug. I uploaded a new patch which should fix it. > > Same URL? I’d try to test that tomorrow then. Yes, I just uploaded a new version of the patch to https://people.freebsd.org/~markj/patches/rtsold_capsicum.diff and would appreciate any further testing that you can do. > >> Also on a running system: > >> > >> root@i386-a3-carp:/ # rm /etc/resolv.conf > >> root@i386-a3-carp:/ # cat /etc/resolv.conf > >> cat: /etc/resolv.conf: No such file or directory > >> root@i386-a3-carp:/ # sh /etc/rc.d/rtsold restart > >> Stopping rtsold. > >> Waiting for PIDS: 1047. > >> Starting rtsold. > >> root@i386-a3-carp:/ # cat /etc/resolv.conf > >> cat: /etc/resolv.conf: No such file or directory > > > > resolvconf -a will only update /etc/resolv.conf if the info in > > /var/run/resolvconf/interfaces/vtnet0 has changed, I believe. Try > > deleting that file too, and then try running rtsol. > > When I deleted /etc/resolv.conf and then rtsol manually it had > re-appeared. Unclear to me what was in /var/run; I just wanted to point > out the difference in behaviour; maybe you are right; I’ll go and > check if deleting in /var/run/ as well makes a difference. I don't observe that behaviour with either the stock or patched rtsol(8): for resolvconf(8) to update /etc/resolv.conf (or re-generate it), something under /var/run/resolvconf/interfaces needs to have changed. So, in my case, deleting /etc/resolv.conf *and* /var/run/resolvconf/interfaces/re0:slaac will cause resolv.conf to be regenerated once rtsold(8) decides to re-run resolvconf(8), but deleting resolv.conf on its own will not. From owner-freebsd-hackers@freebsd.org Wed Oct 24 20:25:53 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1999A1039FB4 for ; Wed, 24 Oct 2018 20:25:53 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DF82786F3; Wed, 24 Oct 2018 20:25:52 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pl1-x635.google.com with SMTP id f8-v6so2760319plb.2; Wed, 24 Oct 2018 13:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=2ioXT9HRsGNXtsENWwjo43ApPJycNSYt1bLCQejCzOI=; b=ukima37bxZoob0xOLLVWoxNrCFNOZCcuQC9lJfQFz6RMRfUhQbFom1GxdrNhtd+ZvH t98GRtnT2Qzwdb79vffcPa5zHMpcFNBKdZq7Cs9vazcwTtDgGbXaWCdkSEcAYeFSgaZf EtpqQu6rmww4FNCUbUGPoRArMOLHry64ND3TGAEaQP7d6hiEGQkpTW0GPxpwWW/zR2R/ ITom+mg2pWHI7gW2ZVdppPpu5KzX1K3EeUZ7Xq8ZEidZjZ0b97EZEaJy/ZNUN1GoksWz ps0TvdgPktoyXxF0ENYqzqeHrE+utQJHF5HdZQqbNF57wqvSRoPbSEaLStg2VW+Jr3Oh c4aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=2ioXT9HRsGNXtsENWwjo43ApPJycNSYt1bLCQejCzOI=; b=rPG5I9N47V7MvGli90CQOQ30DvwgsO2qqBgnmnbB+AOch4Bk9Z3VZKWIq3opvWaaK1 E2ZRJDiGMCceJgYbAE0NIRIKuuookZAgwAjEoNpwu5TvV+pgQSgCduTO3FpcasJXVqpB q4NGFJMS/aPJlEJY6KA07Q2vJf0zJ+9QFCcUtWMaACySsGKtkhlPmSD0i8GrIaXXBbJv 01HeP7XQcXvMHdf/3WrX416PykR400hy0iw3Y8UbcDM+Po/G5nu/8U2oobZWUIXXx+uf WCU92XJMnFs+dGEqf1+jPfV9QAqXNoy7YBdqPxC5MCJD2W9nqxhvKlC089oOPBClHIeE XCRg== X-Gm-Message-State: AGRZ1gIflpm+Q6ZvIyH6SSqpfo0vXISoWJVAE4cIpQfVrXM9Vqw0e2NX nKBRDJMuLzMlYMWnSakJyJpGa/4= X-Google-Smtp-Source: AJdET5fML7MlI5j7zDxByqNYx2reUWQFGoCJOqCDZzuPqlT3BWqjS2OTTcLN//0vi3XVElD13NeWbw== X-Received: by 2002:a17:902:ab8a:: with SMTP id f10-v6mr3849466plr.203.1540412750873; Wed, 24 Oct 2018 13:25:50 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id q127-v6sm20477183pgq.19.2018.10.24.13.25.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 13:25:49 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Millard , Rozhuk Ivan Cc: FreeBSD , Mark Johnston References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> From: Robert Message-ID: Date: Wed, 24 Oct 2018 13:25:48 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 20:25:53 -0000 Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other      6207K Compressed, 54M Uncompressed, 8.96:1 Ratio Swap: 32G Total, 15G Used, 17G Free, 46% Inuse I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? Thanks. On 10/24/18 9:34 AM, Mark Millard wrote: > On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: > >> On Wed, 24 Oct 2018 10:19:20 -0700 >> Robert wrote: >> >>> So the issue is still happening. Please check attached screenshot. >>> The green area is "inactive + cached + free". >>> >>> . . . >> +1 >> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free >> Swap: 112G Total, 19M Used, 112G Free > Just a limited point based on my understanding of "Buf" in > top's display . . . > > If "cached" means "Buf" in top's output, my understanding of Buf > is that it is not a distinct memory area. Instead it totals the > buffer space that is spread across multiple states: Active, > Inactive, Laundry, and possibly Wired(?). > > In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. > If Buf is added to that then there is double counting of > everything included in Buf and the total will be larger > than the TotalMemory. > > Also Inact+Buf+Free may double count some of the Inact space, > the space that happens to be inactive buffer space. > > I may be wrong, but that is my understanding. > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > From owner-freebsd-hackers@freebsd.org Wed Oct 24 20:32:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F9BE106F421 for ; Wed, 24 Oct 2018 20:32:38 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BAA7479149; Wed, 24 Oct 2018 20:32:37 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pl1-x635.google.com with SMTP id bb7-v6so2749105plb.13; Wed, 24 Oct 2018 13:32:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=MhRYysYrJQPyaZBO0+u5UpCPxj6BkJfd30DqXL/pqe4=; b=UQ/F0cqenJ2LdVyEvlq9s5DuXcCitiAy24VkBp49i6cIRV8w77jKWfgK+e9oDK6+ho EGIubslw7LweAUn7fmwlG3FsJA42DymIuXj4lciWDraUsSj4KSrqUeGI6e6aKpT5t6As bG82dJl3V4yW6XQBobOPYfFIUey/HD92GJK4QUbaYikAKrHOE4IreRzW44k/bqOhSEVo 2WP717QzkhfzZDTmetUpgAh69gUSCvyvjbx/9lmSM+uQ9+c5EMWJ1j/u2LqzeKvS2zHp P47usDdg7G33orrbIgIOqr72prixaHRD/ughmCdLakyDwnhr7P/z2xbnbeF3VjFDd/js qKow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=MhRYysYrJQPyaZBO0+u5UpCPxj6BkJfd30DqXL/pqe4=; b=OM0mfPYyWP4ZP5pHNTCqtPRwKaDE8S0xRZk9xVU23bh23MR2gGGAGOLxdZyN/0AeES RZAdDZWiEsU5zQKiFlVoL0VRfza3dX6TC4Bj4COwpBzSP3gwwMiYMstfZab3MuO5wUzA ULH5KCXGx11b60NWr6+HdORsIFZB9IcM9Llrnf9ZocEhMe1OqO+Zn2ovWADm7ITR+UP9 ZueE6o4MAtsPoLXUhi3UVg6A2WacvFZKbW1Jwt6D2kG+ozpYNCAG6gOmtelnUwiGNBuk eQX4sFGHE699SRBv1x9TSSEkzIxMv042yPrQpkzF9xBlmZ/2MVHoMAdvxcStcN5HfHaq ifPw== X-Gm-Message-State: AGRZ1gK4doMHnhM4Lz90BHV8QguZbjE1OEtrE9DtVmL/MsNVx1iVJqXA ZkSMk9HsbzvaOWa8XvMx/xoQdE8= X-Google-Smtp-Source: AJdET5ff9mTUWnIwL3+gdO/MekPhAwbF9K2n/LGN5+xGnvHi4eTgNB4gFhanYaacjYGsc8WR+N/nKw== X-Received: by 2002:a17:902:9698:: with SMTP id n24-v6mr3926595plp.9.1540413156265; Wed, 24 Oct 2018 13:32:36 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id d10-v6sm5407466pgp.26.2018.10.24.13.32.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Oct 2018 13:32:35 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state From: Robert To: Mark Millard , Rozhuk Ivan Cc: FreeBSD , Mark Johnston References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> Message-ID: Date: Wed, 24 Oct 2018 13:32:34 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 20:32:38 -0000 Here is one more from another identical machine: Mem: 1237M Active, 10G Inact, 15G Laundry, 3591M Wired, 846M Free ARC: 81M Total, 869K MFU, 46M MRU, 32K Anon, 491K Header, 34M Other      4664K Compressed, 43M Uncompressed, 9.36:1 Ratio Swap: 32G Total, 1152M Used, 31G Free, 3% Inuse and one unaffected: Mem: 392M Active, 9621M Inact, 136K Laundry, 6106M Wired, 15G Free ARC: 110M Total, 2321K MFU, 54M MRU, 165K Anon, 643K Header, 53M Other      6252K Compressed, 50M Uncompressed, 8.25:1 Ratio Swap: 32G Total, 11M Used, 32G Free All machines are identical and are running same single active process. There are no memory leaks in the process code (at least not through the mmap). Size of all "Active" allocations is roughly 1.5-2GB on all machines. On 10/24/18 11:25 AM, Robert wrote: > Sorry, that wasn't my output, mine (related to the screenshot I've > sent earlier) is: > > Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free > ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other >      6207K Compressed, 54M Uncompressed, 8.96:1 Ratio > Swap: 32G Total, 15G Used, 17G Free, 46% Inuse > > > I'm OK with a low "Free" memory if OS can effectively allocate from > "Inactive", > > but I'm worrying about a sudden move of a huge piece of memory into > "Swap" without any relevant mmap calls. > > > My question is: what else (except mmap) may reduce "Free" memory and > increase "Laundry"\"Swap" in the system? > > Thanks. > > > On 10/24/18 9:34 AM, Mark Millard wrote: >> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: >> >>> On Wed, 24 Oct 2018 10:19:20 -0700 >>> Robert wrote: >>> >>>> So the issue is still happening. Please check attached screenshot. >>>> The green area is "inactive + cached + free". >>>> >>>>   . . . >>> +1 >>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, >>> 617M Free >>> Swap: 112G Total, 19M Used, 112G Free >> Just a limited point based on my understanding of "Buf" in >> top's display . . . >> >> If "cached" means "Buf" in top's output, my understanding of Buf >> is that it is not a distinct memory area. Instead it totals the >> buffer space that is spread across multiple states: Active, >> Inactive, Laundry, and possibly Wired(?). >> >> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. >> If Buf is added to that then there is double counting of >> everything included in Buf and the total will be larger >> than the TotalMemory. >> >> Also Inact+Buf+Free may double count some of the Inact space, >> the space that happens to be inactive buffer space. >> >> I may be wrong, but that is my understanding. >> >> >> === >> Mark Millard >> marklmi at yahoo.com >> ( dsl-only.net went >> away in early 2018-Mar) >> From owner-freebsd-hackers@freebsd.org Wed Oct 24 20:58:28 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1052106FECE for ; Wed, 24 Oct 2018 20:58:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-14.consmr.mail.bf2.yahoo.com (sonic314-14.consmr.mail.bf2.yahoo.com [74.6.132.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 767C07A18B for ; Wed, 24 Oct 2018 20:58:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: xVWS_JIVM1leJvD44d1dGjAG6g0xmlUANppgr8rtEalEJIqfj.JYY_aJv5aY5FZ PuKKhWVIdLdFOXbr64oPgg4f2fdSnqWYRQB_CKM0saf46T6wJRyS73KV6BQm6ISVU1QneHAD6yTc .y2evvce7fbITGkJ0cJKPW3Z9YD7IHvxYxoYA70M66L5KKKbzqhEAWSdYC_XEANcREZAj8XIDzmD 639cFM0Q6.98eup2ToV3k3GL3KCcDdzhdh4FgoOIF4VfvjJqfh7qqvTX4Kw1m0PMs_m.EwyAnEsM PqvBLwgLbJKIuySxROJ9WreQaefY9TaT0CL6iHL4E_I4gz1Uu_IyZ9udng8edIDWWwjqPoRZLxv1 UjO1dM2kcPNQaRy5IvsWZoU7ngubidEaXCgld8Ry_JBUim6RtaGGA0C7qVo36TpLd.YTz_FQjJY8 0eys3xw3Q3t_HCzdB9DXKKi_vI6n.FukmtAaVkHhGb4ZoJi34xdCimRn6K7FKYBae7SMhP89nYk_ soytXOt6QZEDMAE3Qn_VIhdbPRRrYmwV9cn4vXAq1dzVlYkGElOKXYZ_.5ZBlAlk.w7QrNd8ndWI XE67hBJLJ5nW2vIoRnyvzkJKF_.eTHgkyEvNshd.9QsFjERwAf01f1EkwHlvnmJSl67kpKH1FbR9 UwYS4IbUi7t_nxf6lAjvGV4d1HGUGw44OpdSZKEHvIrpEG.UAmD_ZBxODfDczkNVm6uz2RcsfaJ6 eL7XnnkNe54NrF2fhTuxBMy3wXt3YKaFXYupgln95Cf7D1j3e8.OEfTcJN02a_jSOuqRLvQZuWj7 yradKtYOonf4OsBueEnbVp6xJqCPOp3HvNUOkDmnKdr8dpcpZnyECuNfAFEiLoRINV9BQlVD4Gzs OG8oVWiPzYof5lVlZpryAr8IOEHPIE1ojuHMoToVnwgRc_SM.KJExkOQEgqBKG.YDvbpUP2hwrRn XlIXORFIgkAlRY7cs2yPqi33C.PidEFMvjHK2L11ysnpDX3NR0CP9NCUZC4LpA_O8yWUSRacv4G6 4eZtdlVRWS_IsVXwPxjw.KpVrNVv0.sZ_OB6uquc1SnVD9HB4GA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.bf2.yahoo.com with HTTP; Wed, 24 Oct 2018 20:58:20 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp413.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 2849920b5d416d57c857dadf0431959f; Wed, 24 Oct 2018 20:58:17 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Sudden grow of memory in "Laundry" state From: Mark Millard In-Reply-To: Date: Wed, 24 Oct 2018 13:58:14 -0700 Cc: Rozhuk Ivan , FreeBSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> To: Robert X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 20:58:28 -0000 On 2018-Oct-24, at 1:25 PM, Robert = wrote: > Sorry, that wasn't my output, mine (related to the screenshot I've = sent earlier) is: No screen shot made it through the list back out to those that get messages from the freebsd-hackers at freebsd.org reference in the CC. The list limits itself to text as I understand. > Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free > ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other > 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio > Swap: 32G Total, 15G Used, 17G Free, 46% Inuse Relative to my limited point: I do not know the status of mutually-exclusive categorizations vs. not for ZFS ARC and Mem. Unfortunately, as I understand things, it is questionable if adding -S to the top command gives you swap information that can point to what makes up the 15G swapped out by totaling the sizes listed. But you might at least be able to infer what processes became swapped out even if you can not get a good size for the swap space use for each. Using -ores does seem like it puts the top users of resident memory at the top of top's process list. Sufficient Active RAM use by processes that stay active will tend to cause inactive processes to be swapped out. FreeBSD does not swap out processes that stay active: it pages those as needed instead of swapping. So using top -Sores might allow watching what active process(es) grow and stay active and what inactive processes are swapped out at the time of the activity. I do infer that the 15G Used for Swap is tied to processes that were not active when swapped out. > I'm OK with a low "Free" memory if OS can effectively allocate from = "Inactive", >=20 > but I'm worrying about a sudden move of a huge piece of memory into = "Swap" without any relevant mmap calls. >=20 >=20 > My question is: what else (except mmap) may reduce "Free" memory and = increase "Laundry"\"Swap" in the system? >=20 > Thanks. >=20 >=20 > On 10/24/18 9:34 AM, Mark Millard wrote: >> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan = wrote: >>=20 >>> On Wed, 24 Oct 2018 10:19:20 -0700 >>> Robert wrote: >>>=20 >>>> So the issue is still happening. Please check attached screenshot. >>>> The green area is "inactive + cached + free". >>>>=20 >>>> . . . >>> +1 >>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, = 617M Free >>> Swap: 112G Total, 19M Used, 112G Free >> Just a limited point based on my understanding of "Buf" in >> top's display . . . >>=20 >> If "cached" means "Buf" in top's output, my understanding of Buf >> is that it is not a distinct memory area. Instead it totals the >> buffer space that is spread across multiple states: Active, >> Inactive, Laundry, and possibly Wired(?). >>=20 >> In other words: TotalMemory =3D Active+Inact+Laundry+Wired+Free. >> If Buf is added to that then there is double counting of >> everything included in Buf and the total will be larger >> than the TotalMemory. >>=20 >> Also Inact+Buf+Free may double count some of the Inact space, >> the space that happens to be inactive buffer space. >>=20 >> I may be wrong, but that is my understanding. >>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Wed Oct 24 21:54:35 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B2AE10714F2 for ; Wed, 24 Oct 2018 21:54:35 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: from mail-lf1-x143.google.com (mail-lf1-x143.google.com [IPv6:2a00:1450:4864:20::143]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B62AD7BB33; Wed, 24 Oct 2018 21:54:34 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: by mail-lf1-x143.google.com with SMTP id n3-v6so5180977lfe.7; Wed, 24 Oct 2018 14:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:date:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Sj6F8dCT1yxo5eVISkVkmV/tk/QF4CNmmmlJjANdqPo=; b=rtmpeyUFvK+pfLgukcwmit6ihYSmQ+qSoIRVPI3HYjV5IK+J+QsLMMfarqpfN8uZCb wgTwGoa67HFNYUoQsCwbYWWLeXmBcdItBdxAxdwd7POnbDl5K855Ca3e5kllRAfyi0nm l8SyzQaibc93aa81ER+TltKYMFxEEJgYQhBKxQWzYkJsXuNLDsa/tm5H/iLmodTOhpTY EJ+N6HAxEmohOzUlXs/Gcx3FIEb3QWlPpJO76ebYOXWSbSN0ucXJFVwVo4K3SOKAzaES uzlmGcG+FoQNDSmML3vmgzoy+Q7jJNoTBBEqeQod5/H6AixiXBeo0BdHzODRzL/64DMM E4uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Sj6F8dCT1yxo5eVISkVkmV/tk/QF4CNmmmlJjANdqPo=; b=hAdsO7hjYNwCX6yjdSp1x6OZlHWTcbzOEsedH0p54RA4Klp9VXXg6dmCO3qy+Bitez biKzdkyDUxRgQkdBvPzwTfA6PLEDn40mvl7GztfwELru4Vz7io3GkUAuOEQGo8PJnX4w CySxHItn9iwT/Q4o0qvB2nUvNzy0sz8VRe67ff52Tt5OR9eWrJ9J2F88aKJIQ4iQQS29 BdAQu2tnXl4kEwStItZXklzZD3mQGSia+tkl4j6pj7ISxmlcnnnWthh1e+IL0meHkscY 1hHaj160SaAFU+//6JLyflio9qbRlCIH48w9iQM3/mazYZgNIdpGKbNJsbFUXl6b4YYW AGNA== X-Gm-Message-State: AGRZ1gLypLV32szM+BUpXe9i2HguLm0mihiHLcwlmRsKcN75cwupTU0M /xwCspMpjUvp8hpGn7GZT28= X-Google-Smtp-Source: AJdET5fb6yFR1KBCW6uRHUNF+tKPS0MHzX3CGlzPC+WPChMSXT1zTE9kpv8OIkiLflqlvIJnDb33lA== X-Received: by 2002:a19:c396:: with SMTP id t144mr56021lff.110.1540418073091; Wed, 24 Oct 2018 14:54:33 -0700 (PDT) Received: from localhost ([2001:470:1f15:3d8:7285:c2ff:fe37:5722]) by smtp.gmail.com with ESMTPSA id z10-v6sm399056lfj.50.2018.10.24.14.54.31 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Oct 2018 14:54:31 -0700 (PDT) From: Rozhuk Ivan X-Google-Original-From: Rozhuk Ivan Date: Thu, 25 Oct 2018 00:54:29 +0300 To: Robert Cc: Mark Millard , FreeBSD , Mark Johnston Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <20181025005429.597e9066@gmail.com> In-Reply-To: References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 21:54:35 -0000 On Wed, 24 Oct 2018 13:25:48 -0700 Robert wrote: > My question is: what else (except mmap) may reduce "Free" memory and > increase "Laundry"\"Swap" in the system? > FreeBSD mem usage is at least "strange". I got 32GB installed, sum of RSS runnig programs ~16GB. Why system use swap? If there is no swap then system some times freezes because cant allocate mem. WTF!? Before FreeBSD I use win 7 on system with 16GB, and same software: firefox + vbox. I do not use swap, and in same use cases I got ~half free memory. Probably now same software consume more mem, let x2 = 16BG, but where is another 16GB free ram!? From owner-freebsd-hackers@freebsd.org Fri Oct 26 16:19:21 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B061010EACC8 for ; Fri, 26 Oct 2018 16:19:21 +0000 (UTC) (envelope-from olli@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C8C77D1B3 for ; Fri, 26 Oct 2018 16:19:21 +0000 (UTC) (envelope-from olli@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1154) id 4264C18240; Fri, 26 Oct 2018 16:19:21 +0000 (UTC) To: freebsd-hackers@freebsd.org Subject: Re: is svnews down or just gone Message-Id: <20181026161921.4264C18240@freefall.freebsd.org> Date: Fri, 26 Oct 2018 16:19:21 +0000 (UTC) From: Oliver Fromme X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2018 16:19:21 -0000 Mark Saad wrote (2018-06-05 20:44:16 UTC): > Does anyone know what happened to http://www.secnetix.de/olli/FreeBSD/svnews/ > I enjoyed reading the summary of commits. Is there anything else like > this out there ? I'm not working at Secnetix anymore, so I had to move my web tools "SVNews" and "Porgle" to a new site. These are the new addresses: SVNews: http://inof.de/FreeBSD/svnews/ Porgle: http://inof.de/FreeBSD/porgle/ For those who don't know them: SVNews shows a summary of the latest commits to the FreeBSD base (not ports), optionally with diffs. Porgle is a search engine for the Ports Collection that supports regular expressions, searching the long descriptions and package lists (plist). If you have any questions or encounter any problems, please mail me directly. I'm currently *not* subscribed to -hackers. Best regards - Olli From owner-freebsd-hackers@freebsd.org Fri Oct 26 22:07:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D00E310C973A for ; Fri, 26 Oct 2018 22:07:12 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 38E626D294; Fri, 26 Oct 2018 22:07:12 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pl1-x62f.google.com with SMTP id 30-v6so1087255plb.10; Fri, 26 Oct 2018 15:07:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=jbjE8CdogO6iBzRNmWLN9ZkQlhu+jpKarMAv7prj3WQ=; b=HDuLhiHNvmhxmJQcLCCA+IfJPQy9XYWnlhZOI4QGZrEoOr+H09aQkVVv/tZV+HLdLB opB8mYY/HM0RB1+e0mIikVxqvVbd/Fu3nk/6TLaoyvKwlkU0v+AghNQ3bifp/XLgSDgt LAjvHZfWs/8UXtIxXLU+AR8Oh7aZujgyeoXaupEmy4+c1XW3VZ7e9TbAV6EMiY2GPbsp DkISKRcqSyWWoC2uK/fPG9hgf2eZueQbvRrqCQ5HYkuLhB7GSvFFr/WSrajpwSrDc46G wzFwUmbb7TpL0pdPzcYPGjvrsNZd0XqWwWWxEixMdr019CIWwULHKmTXW6wqX4wlFspm 2nWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=jbjE8CdogO6iBzRNmWLN9ZkQlhu+jpKarMAv7prj3WQ=; b=AYoTFLdfzv7dmv4qBg7tnqKTqTuhzlkOMvyuKUHEZxDY2pXNhZPw57KU8m+R4AFu4v wg3CyaRx8l8Yt/BwF4R+nHQN1YXHCcjmP1GVlmk+Vm/ynT4ulaCW+SvEdK9oYjpTfYyE E8aAqDaNhncdrvv8qTXG//mfAF2S7KC0lC+0LKNVoHGKNsNeO8nJmRZTxI1bNhuuGUKE TAmwyp0adYY2EsgQhdc3pf9p4vHZHVErnGIRT5MOOcJ+t2HWbNlfB3/sQhuVAt2rr/RX gcvtCfXs5ed8VrkK9nxru9PVYcaPSyuiQM7iB5Jv0DLDeMfZXjtJfYNpQuwywuSE9kot Ewbg== X-Gm-Message-State: AGRZ1gLXCRh7cx+Lmu8dsNgzEqSHyWYGGjrWXju4tqBg/zvrSoxip+cD 00pLkqnd/gr60Xl7MiLd384ZtKM= X-Google-Smtp-Source: AJdET5dRHYMR6c9lXJoKHttrcO84w9CRx0cbFG2IMjmvrwcxkHZQHKDLNYVkm0/5o1VjHl7QVC/f0w== X-Received: by 2002:a17:902:e101:: with SMTP id cc1-v6mr5224088plb.165.1540591629513; Fri, 26 Oct 2018 15:07:09 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id p63-v6sm1946329pfg.79.2018.10.26.15.07.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Oct 2018 15:07:08 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Millard Cc: Rozhuk Ivan , FreeBSD , Mark Johnston References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> From: Robert Message-ID: <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> Date: Fri, 26 Oct 2018 15:07:07 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2018 22:07:13 -0000 Sorry, let me be more specific. Please look into: https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing (wait until charts fully loaded). These are all memory states and mmap\munmap stats collected. Y axis is in MBs, X is a timeline. It's not a problem to understand which process produces allocations and is being swapped. I know this for sure. The issue is: I strongly believe that by some reason FreeBSD kernel fails to reuse deallocated memory properly. Looking into graphs we can see following: 1. When process allocates memory (mmap), "Active" memory increases, "Free" memory decreases (that's expected). 2. When process deallocates memory (munmap), "Inactive" memory increases, "Active" memory decreases. Memory never returns into "Free" state. That's kind of expected as well. 3. At some point, when sum of "Active" and "Inactive" memory exceeds some upper memory limits, OS starts to push "Inactive" memory into "Laundry" and "Swap". This happens very quick and unexpectedly. Now why OS doesn't reuse huge amounts of "Inactive" memory when calling mmap? Or my assumption about availability of "Inactive" memory is wrong? Which one is free for allocations then? Thanks. On 10/24/18 11:58 AM, Mark Millard wrote: > On 2018-Oct-24, at 1:25 PM, Robert wrote: > >> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: > No screen shot made it through the list back out to those that > get messages from the freebsd-hackers at freebsd.org reference > in the CC. The list limits itself to text as I understand. > >> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free >> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other >> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse > Relative to my limited point: I do not know the status of > mutually-exclusive categorizations vs. not for ZFS ARC and > Mem. > > Unfortunately, as I understand things, it is questionable if > adding -S to the top command gives you swap information that > can point to what makes up the 15G swapped out by totaling > the sizes listed. But you might at least be able to infer > what processes became swapped out even if you can not get > a good size for the swap space use for each. > > Using -ores does seem like it puts the top users of resident > memory at the top of top's process list. > > Sufficient Active RAM use by processes that stay active will > tend to cause inactive processes to be swapped out. FreeBSD > does not swap out processes that stay active: it pages those > as needed instead of swapping. > > So using top -Sores might allow watching what active > process(es) grow and stay active and what inactive processes > are swapped out at the time of the activity. > > I do infer that the 15G Used for Swap is tied to processes > that were not active when swapped out. > >> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", >> >> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. >> >> >> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? >> >> Thanks. >> >> >> On 10/24/18 9:34 AM, Mark Millard wrote: >>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: >>> >>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>> Robert wrote: >>>> >>>>> So the issue is still happening. Please check attached screenshot. >>>>> The green area is "inactive + cached + free". >>>>> >>>>> . . . >>>> +1 >>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free >>>> Swap: 112G Total, 19M Used, 112G Free >>> Just a limited point based on my understanding of "Buf" in >>> top's display . . . >>> >>> If "cached" means "Buf" in top's output, my understanding of Buf >>> is that it is not a distinct memory area. Instead it totals the >>> buffer space that is spread across multiple states: Active, >>> Inactive, Laundry, and possibly Wired(?). >>> >>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. >>> If Buf is added to that then there is double counting of >>> everything included in Buf and the total will be larger >>> than the TotalMemory. >>> >>> Also Inact+Buf+Free may double count some of the Inact space, >>> the space that happens to be inactive buffer space. >>> >>> I may be wrong, but that is my understanding. >>> > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > From owner-freebsd-hackers@freebsd.org Sat Oct 27 02:40:19 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D17BD10D3687 for ; Sat, 27 Oct 2018 02:40:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-23.consmr.mail.ne1.yahoo.com (sonic311-23.consmr.mail.ne1.yahoo.com [66.163.188.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6503E777C1 for ; Sat, 27 Oct 2018 02:40:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ENZqis4VM1muohFrYRsdxktG.CGiVOZ1iByxt2Nv0txVEIiJqMmjuo.K_s6LzcK wJ_KhWaimixnCrrdPAjb6mnFLTIpKDIOEeDhvrCMtss3oLOFRzlac3R4kr_TdeA6GCIjuu.mXN3p UC24z2DJ4gSx0YkXaZ9zjUmvpAYaR.vBFOK8pZsbXZp1dH1VgI.iuMsXwfuJeLQ0ht6VnfwSG9.v pBfhnZ1dFXdoIBC1Rxls0g2q2f9eVR2F5c1iDeHDk_GlsOcJfgqIVgNz9skzvPEhv6BaGWMvz2AV 3Vg3aIbOROCX3W5JIDm_I71eqfvtD3UEarnuI0OkwQp1BYDP58_7WzXBVCHQrC345WaoXeMuzSp9 .qc1B9gYTIvnbQXc5UOOdeJ5SoeR8VBX.psqsQYeaiopgKqCXnSE2U_QBZ_ZLmNkVxAfJ1U2dJsN ZNH91gmyZDXkyvB2E08UZQroOQ4oD6RAKJuMWO8bBX6WDyjb4dVP4S_0xURt1yPRvaUdhIJ8MGf. YvYnQEsndVxwromNLzO02oTRaEH87x4UVeFjNxAo903BOGzStVHdPq.XaCjIU17aCiRxaXX9M_IN G7DxNKo8ol6g6Jl885_jsJgIcvt.Kt6T5_JtIKL5M4Zcct5.B9D4E5W9eHxHk0hZQ3e1YmNA5yDn feeb4I0Tluh2QwnawuOAtdXNGRBh390Y6v3jeev8Ed_W9_3y_lDqc7oG_EPun190XVWnUp0DoKJX 9LbXKtHXGpPAil9zl5A4ciE8BO9GvqB3xNt2jX5iwYVNFZQT58wzN.lKJFHoRIzDnZxFi3yVxPo7 ZdclVjcuIhN4ZAoKc4.2jF2TqQMNLqtqcxYcTz1GmgS_bC2g1UWw78fpuP537bLSJHFl_keo_J6Y 3Iof.qolztsJWkpAEqsxLosl_9PfrOvOWzQqXWnJFJpMSSe3VMyQudqW7kvqg0fM7..8riud0C8J mK4xKEQb86QYUNtCyPNbKm7bECnp.YooQpgNNz5XUiXPsWq4QCop7uV06n8Pm2VKsFTx_9L0kfgA ufYh_qElQy1CD24YAFKMsR8b5sBCB_V7f2hVaeyN3 Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.ne1.yahoo.com with HTTP; Sat, 27 Oct 2018 02:40:12 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp430.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID da4a79df75af908369c523e799701d0e; Sat, 27 Oct 2018 02:40:10 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Sudden grow of memory in "Laundry" state From: Mark Millard In-Reply-To: <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> Date: Fri, 26 Oct 2018 19:40:09 -0700 Cc: Rozhuk Ivan , FreeBSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> To: Robert X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 02:40:20 -0000 On 2018-Oct-26, at 3:07 PM, Robert = wrote: > Sorry, let me be more specific. >=20 > Please look into: = https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4Khcvd= pOL68gEUM/edit?usp=3Dsharing (wait until charts fully loaded). Thanks for giving folks access to the charts originally referred to. > These are all memory states and mmap\munmap stats collected. Y axis is = in MBs, X is a timeline. Some things folks looking into this might want to know: MAP_PRIVATE in use? Vs.: MAP_SHARED in use? MAP_NOSYNC in use or not? MAP_ANON in use or not? MAP_FIXED in use or not? (Probably not?) But I cover MAP_NOSYNC and another option that is in a 3rd call below and do not need such information for what I've written. > It's not a problem to understand which process produces allocations = and is being swapped. I know this for sure. >=20 > The issue is: I strongly believe that by some reason FreeBSD kernel = fails to reuse deallocated memory properly. >=20 > Looking into graphs we can see following: >=20 > 1. When process allocates memory (mmap), "Active" memory increases, = "Free" memory decreases (that's expected). >=20 > 2. When process deallocates memory (munmap), "Inactive" memory = increases, "Active" memory decreases. >=20 > Memory never returns into "Free" state. That's kind of expected as = well. =46rom the description of MAP_NOSYNC for mmap (vs. the flushing behavior): . . . Without this option any VM pages you dirty may be flushed to disk every so = often (every 30-60 seconds usually) which can create = perfor- mance problems if you do not need that to occur = (such as when you are using shared file-backed mmap = regions for IPC purposes). Dirty data will be flushed = auto- matically when all mappings of an object are = removed and all descriptors referencing the object are = closed. Note that VM/file system coherency is maintained whether you use MAP_NOSYNC or not. Note the specified behavior for flushing out "dirty data" unless MAP_NOSYNC is in use. (I note another alternative later.) As I understand it FreeBSD uses the swapping/paging code to do the flush activity: part of the swap/page space is mapped into the the file in question and the flushing is a form of swapping/paging out pages. [Note: Top does not keep track of changes in swap space, for example a "swapon" done after top has started displaying things will not show an increased swap total but the usage can show larger than the shown total. Flushing out to a mapped file might be an example of this for all I know.] Also reported for flushing behavior is: . . . The fsync(2) system call will flush all dirty data and metadata associated with a file, including dirty NOSYNC VM data, to physical media. The sync(8) command and sync(2) system call generally do not flush dirty NOSYNC VM data. The msync(2) system call is usually not needed since BSD implements a = coherent file system buffer cache. However, it may be = used to associate dirty VM pages with file system = buffers and thus cause them to be flushed to physical media = sooner rather than later. As for munmap: its description is that the address range is still reserved afterwards, quoting the description: The munmap () system call deletes the mappings and guards for the speci- fied address range, and causes further references to addresses = within the range to generate invalid memory references. That last is not equivalent to the address range being "free" in that the range still counts against the process address space. (So being accurate about what is about RAM availability vs. address space usage/availability is important in order to avoid confusions.) It would appear that to force invalid memory references involves keeping page descriptions around but they would be inactive, rather than active. This is true no matter if RAM is still associated or not. (So this could potentially lead to a form of extra counting of RAM use, sort of like in my original note.) See later below for another means of control . . . Remember: "Dirty data will be flushed automatically when all mappings of an object are removed and all descriptors referencing the object are closed". So without MAP_NOSYNC the flushing is expected. But see below for another means of control . . . There is another call madvise that has an option tied to enabling freeing pages and avoiding flushing the pages: MADV_FREE Gives the VM system the freedom to free pages, and = tells the system that information in the specified page = range is no longer important. This is an efficient way = of allowing malloc(3) to free pages anywhere in the = address space, while keeping the address space valid. The = next time that the page is referenced, the page might = be demand zeroed, or might contain the data that was = there before the MADV_FREE call. References made to = that address space range will not make the VM system = page the information back in from backing store until the = page is modified again. This is a way to let the system free page ranges and allow later use of the address range in the process's address space. No descriptions of page ranges that should generate invalid memory references, so no need of such "inactive pages". MADV_FREE makes clear that your expectations of the meaning of munmap does not seem to match FreeBSD's actual usage: MADV_FREE must be explicit to get the behavior you appear to be looking for. At least that is the way I read the documentation's meaning. MAP_NOSYNC does not seem sufficient for matching what you are looking for as the behavioral properties --but it appears possibly necessary up to when MADV_FREE can be used. > 3. At some point, when sum of "Active" and "Inactive" memory exceeds = some upper memory limits, >=20 > OS starts to push "Inactive" memory into "Laundry" and "Swap". This = happens very quick and unexpectedly. This is the flushing activity documented as far as I can tell. > Now why OS doesn't reuse huge amounts of "Inactive" memory when = calling mmap? Without MADV_FREE use the system does not have "the freedom to free pages". Without MAP_NOSYNC as well it is expected to flush out some pages at various times as things go along. > Or my assumption about availability of "Inactive" memory is wrong? = Which one is free for allocations then? Pages that are inactive and dirty, normally have to be flushed out before the RAM for the page can be freed for other uses. MADV_FREE is for indicating when this is not the case and the usage of the RAM has reach a stage where the RAM can be more directly freed (no longer tied to the process). At least that is my understanding. Mark Johnston had already written about MADV_FREE but not with such quoting of related documentation. If he and I seem to contradict each other anywhere, believe Mark J. I'm no FreeBSD expert. I'm just trying to reference and understand the documentation. > Thanks. >=20 >=20 > On 10/24/18 11:58 AM, Mark Millard wrote: >> On 2018-Oct-24, at 1:25 PM, Robert = wrote: >>=20 >>> Sorry, that wasn't my output, mine (related to the screenshot I've = sent earlier) is: >> No screen shot made it through the list back out to those that >> get messages from the freebsd-hackers at freebsd.org reference >> in the CC. The list limits itself to text as I understand. >>=20 >>> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free >>> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M = Other >>> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >>> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse >> Relative to my limited point: I do not know the status of >> mutually-exclusive categorizations vs. not for ZFS ARC and >> Mem. >>=20 >> Unfortunately, as I understand things, it is questionable if >> adding -S to the top command gives you swap information that >> can point to what makes up the 15G swapped out by totaling >> the sizes listed. But you might at least be able to infer >> what processes became swapped out even if you can not get >> a good size for the swap space use for each. >>=20 >> Using -ores does seem like it puts the top users of resident >> memory at the top of top's process list. >>=20 >> Sufficient Active RAM use by processes that stay active will >> tend to cause inactive processes to be swapped out. FreeBSD >> does not swap out processes that stay active: it pages those >> as needed instead of swapping. >>=20 >> So using top -Sores might allow watching what active >> process(es) grow and stay active and what inactive processes >> are swapped out at the time of the activity. >>=20 >> I do infer that the 15G Used for Swap is tied to processes >> that were not active when swapped out. >>=20 >>> I'm OK with a low "Free" memory if OS can effectively allocate from = "Inactive", >>>=20 >>> but I'm worrying about a sudden move of a huge piece of memory into = "Swap" without any relevant mmap calls. >>>=20 >>>=20 >>> My question is: what else (except mmap) may reduce "Free" memory and = increase "Laundry"\"Swap" in the system? >>>=20 >>> Thanks. >>>=20 >>>=20 >>> On 10/24/18 9:34 AM, Mark Millard wrote: >>>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan = wrote: >>>>=20 >>>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>>> Robert wrote: >>>>>=20 >>>>>> So the issue is still happening. Please check attached = screenshot. >>>>>> The green area is "inactive + cached + free". >>>>>>=20 >>>>>> . . . >>>>> +1 >>>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M = Buf, 617M Free >>>>> Swap: 112G Total, 19M Used, 112G Free >>>> Just a limited point based on my understanding of "Buf" in >>>> top's display . . . >>>>=20 >>>> If "cached" means "Buf" in top's output, my understanding of Buf >>>> is that it is not a distinct memory area. Instead it totals the >>>> buffer space that is spread across multiple states: Active, >>>> Inactive, Laundry, and possibly Wired(?). >>>>=20 >>>> In other words: TotalMemory =3D Active+Inact+Laundry+Wired+Free. >>>> If Buf is added to that then there is double counting of >>>> everything included in Buf and the total will be larger >>>> than the TotalMemory. >>>>=20 >>>> Also Inact+Buf+Free may double count some of the Inact space, >>>> the space that happens to be inactive buffer space. >>>>=20 >>>> I may be wrong, but that is my understanding. >>>>=20 >=20 From owner-freebsd-hackers@freebsd.org Sat Oct 27 05:36:23 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89D7910DA03C for ; Sat, 27 Oct 2018 05:36:23 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E87ED7E8AD; Sat, 27 Oct 2018 05:36:22 +0000 (UTC) (envelope-from robert.ayrapetyan@gmail.com) Received: by mail-pf1-x434.google.com with SMTP id l17-v6so1530627pff.2; Fri, 26 Oct 2018 22:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=FNbKgPUjlHu+vGyWlUasHs5HqjDIfI3pdQeNQMlhJXI=; b=nXsbgddzXxkSv/y3sTcf8b42epzsgUmmK9qrZfAVLI/BG25Gy4ERnJN5dgALIFe9RS 0hGQMVeSsnjesQGt3p3A5J9YU3O1BbYPRypt6o6F8or7O4OCoSetJpct2uVPLsZrN9TL YZkmMGnX/KpiK7NOKDkD74+fHrrf8zYfGxCsOS4SLQb1OrcPIpPn9GGcLq+R76sabVjh 2bRrFPcwRqbpsYdFGmcEihK0jz8Dox3Cd0hKVZvzsh41ccf7v1ashc30xU69zIXBMmMr Y4d42FaSMUcGkXMcUQk/G1nkhCs+255cPpI1ukT2oZIPOE1KH4q6wvfH6HGYZQXwDPGs yB5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=FNbKgPUjlHu+vGyWlUasHs5HqjDIfI3pdQeNQMlhJXI=; b=fGgEFORlL2J3OJHOXXQIodC/K1CLe0A2Mo/L2PAjy0Ln09eeDZGG0lH7pj7dTnmksh u8optynI7z9FJDFHKWDpQd8kYjtfXJSdTpgErLHDwdfgKfl1qFyMDRfpsBGXWj9JmN9O cb34qf8EjEdM9iDZjo1HDm4mr2wYVsc25i4jUYuLXCiKef/v6r0d4jmiLiWidMUuxOkK LhUTbSWPH4Ptrdda5dllSbFSXwZXNHodOFqu747Q05ndeoetNvzBnp7S6F/GPgC/U74f IApn3gY7gLb3p69xRtuBZFGx3P6zcn2l/CIFhg6v7MDbjHFI22zxRKE3toBvtPmjHxfR yKMQ== X-Gm-Message-State: AGRZ1gLztg/7qBU4wjY+FzqRc7FkE/qizrimOgKLH0urtKmSKnNIVYP6 OPYEfnZPRj5X3UZiaI5DxjAQzP8= X-Google-Smtp-Source: AJdET5d1GQd6VJjAJO6p2C8+n1LPEMBnszXG136JG18fq3eGI7x24c5msIv9NjOBFcN3Ut2iXivWkQ== X-Received: by 2002:a63:4384:: with SMTP id q126mr6046872pga.160.1540618580857; Fri, 26 Oct 2018 22:36:20 -0700 (PDT) Received: from [192.168.1.113] (c-73-202-42-181.hsd1.ca.comcast.net. [73.202.42.181]) by smtp.gmail.com with ESMTPSA id q7-v6sm8649741pgv.78.2018.10.26.22.36.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Oct 2018 22:36:19 -0700 (PDT) Subject: Re: Sudden grow of memory in "Laundry" state To: Mark Millard Cc: Rozhuk Ivan , FreeBSD , Mark Johnston References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> From: Robert Message-ID: Date: Fri, 26 Oct 2018 22:36:18 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 05:36:23 -0000 Hi Mark, thanks for you reply. Regarding memory flags - I'm not setting them directly anywhere. Initially my app allocates shared memory segment of size RAM \ 2: namespace bip =boost::interprocess; bip::managed_shared_memory(bip::open_or_create,SHMEM_SEG_NAME, size) and never resizes\deallocates it. Later it uses only "new"-s and "mallocs". When I watched for flags in DTRACE, they were all like below: 2018 Oct 20 11:24:48 mmap args: addr:0 len:2097152 prot:3 flags:1002 fd:-1 offset:0, which is: #define MAP_PRIVATE 0x0002 /* changes are private */ and #define MAP_ANON 0x1000 /* allocated from memory, swap space */ This is what alloc\new generates by default. Btw, if you look into mmap\munmap sizes you will notice it generates 3-5 GB more mmaps than munmaps every 5 minutes. This means machine should be out of memory within an hour. But it doesn't happen so fast... Why DTRACE lies here? Or is it because DTRACE can't catch all munmaps due to optimizations in kernel code, which was discussed recently in mailing list? Anyway, from your response I understood that using MADV_FREE may help. Any idea of how to use it properly? Should I call madvise after every free\delete on the "freed" pages? This sounds like something completely wrong... On 10/26/18 5:40 PM, Mark Millard wrote: > On 2018-Oct-26, at 3:07 PM, Robert wrote: > >> Sorry, let me be more specific. >> >> Please look into: https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing (wait until charts fully loaded). > Thanks for giving folks access to the charts originally referred to. > >> These are all memory states and mmap\munmap stats collected. Y axis is in MBs, X is a timeline. > Some things folks looking into this might want to know: > > MAP_PRIVATE in use? Vs.: MAP_SHARED in use? > > MAP_NOSYNC in use or not? > > MAP_ANON in use or not? > > MAP_FIXED in use or not? (Probably not?) > > But I cover MAP_NOSYNC and another option that is > in a 3rd call below and do not need such information > for what I've written. > >> It's not a problem to understand which process produces allocations and is being swapped. I know this for sure. >> >> The issue is: I strongly believe that by some reason FreeBSD kernel fails to reuse deallocated memory properly. >> >> Looking into graphs we can see following: >> >> 1. When process allocates memory (mmap), "Active" memory increases, "Free" memory decreases (that's expected). >> >> 2. When process deallocates memory (munmap), "Inactive" memory increases, "Active" memory decreases. >> >> Memory never returns into "Free" state. That's kind of expected as well. > From the description of MAP_NOSYNC for mmap > (vs. the flushing behavior): > > . . . Without this option any VM > pages you dirty may be flushed to disk every so often > (every 30-60 seconds usually) which can create perfor- > mance problems if you do not need that to occur (such > as when you are using shared file-backed mmap regions > for IPC purposes). Dirty data will be flushed auto- > matically when all mappings of an object are removed > and all descriptors referencing the object are closed. > Note that VM/file system coherency is maintained > whether you use MAP_NOSYNC or not. > > Note the specified behavior for flushing out "dirty data" > unless MAP_NOSYNC is in use. (I note another alternative > later.) > > As I understand it FreeBSD uses the swapping/paging code to do the > flush activity: part of the swap/page space is mapped into the > the file in question and the flushing is a form of swapping/paging > out pages. > > [Note: Top does not keep track of changes in swap space, > for example a "swapon" done after top has started > displaying things will not show an increased swap total > but the usage can show larger than the shown total. > Flushing out to a mapped file might be an example of > this for all I know.] > > Also reported for flushing behavior is: > > . . . The fsync(2) system call will flush all dirty data and > metadata associated with a file, including dirty > NOSYNC VM data, to physical media. The sync(8) > command and sync(2) system call generally do not > flush dirty NOSYNC VM data. The msync(2) system > call is > usually not needed since BSD implements a coherent > file system buffer cache. However, it may be used to > associate dirty VM pages with file system buffers and > thus cause them to be flushed to physical media sooner > rather than later. > > > As for munmap: its description is that the address range is still > reserved afterwards, quoting the description: > > The munmap () system call deletes the mappings and guards for the speci- > fied address range, and causes further references to addresses within the > range to generate invalid memory references. > > That last is not equivalent to the address range being "free" > in that the range still counts against the process address space. > (So being accurate about what is about RAM availability vs. address > space usage/availability is important in order to avoid confusions.) > > It would appear that to force invalid memory references involves > keeping page descriptions around but they would be inactive, > rather than active. This is true no matter if RAM is still associated > or not. (So this could potentially lead to a form of extra counting > of RAM use, sort of like in my original note.) See later below for > another means of control . . . > > Remember: "Dirty data will be flushed automatically when all mappings of > an object are removed and all descriptors referencing the object are > closed". So without MAP_NOSYNC the flushing is expected. But see below > for another means of control . . . > > There is another call madvise that has an option tied > to enabling freeing pages and avoiding flushing the > pages: > > MADV_FREE Gives the VM system the freedom to free pages, and tells > the system that information in the specified page range > is no longer important. This is an efficient way of > allowing malloc(3) to free pages anywhere in the address > space, while keeping the address space valid. The next > time that the page is referenced, the page might be > demand zeroed, or might contain the data that was there > before the MADV_FREE call. References made to that > address space range will not make the VM system page the > information back in from backing store until the page is > modified again. > > This is a way to let the system free page ranges and > allow later use of the address range in the process's > address space. No descriptions of page ranges that should > generate invalid memory references, so no need of such > "inactive pages". > > MADV_FREE makes clear that your expectations of the meaning > of munmap does not seem to match FreeBSD's actual usage: > MADV_FREE must be explicit to get the behavior you appear > to be looking for. At least that is the way I read the > documentation's meaning. MAP_NOSYNC does not seem sufficient > for matching what you are looking for as the behavioral > properties --but it appears possibly necessary up to when > MADV_FREE can be used. > >> 3. At some point, when sum of "Active" and "Inactive" memory exceeds some upper memory limits, >> >> OS starts to push "Inactive" memory into "Laundry" and "Swap". This happens very quick and unexpectedly. > This is the flushing activity documented as far as I can tell. > >> Now why OS doesn't reuse huge amounts of "Inactive" memory when calling mmap? > Without MADV_FREE use the system does not have "the freedom > to free pages". Without MAP_NOSYNC as well it is expected > to flush out some pages at various times as things go along. > >> Or my assumption about availability of "Inactive" memory is wrong? Which one is free for allocations then? > Pages that are inactive and dirty, normally have to be > flushed out before the RAM for the page can be freed > for other uses. MADV_FREE is for indicating when this is > not the case and the usage of the RAM has reach a stage > where the RAM can be more directly freed (no longer tied > to the process). > > At least that is my understanding. > > Mark Johnston had already written about MADV_FREE but not > with such quoting of related documentation. If he and I > seem to contradict each other anywhere, believe Mark J. > I'm no FreeBSD expert. I'm just trying to reference and > understand the documentation. > >> Thanks. >> >> >> On 10/24/18 11:58 AM, Mark Millard wrote: >>> On 2018-Oct-24, at 1:25 PM, Robert wrote: >>> >>>> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: >>> No screen shot made it through the list back out to those that >>> get messages from the freebsd-hackers at freebsd.org reference >>> in the CC. The list limits itself to text as I understand. >>> >>>> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free >>>> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other >>>> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >>>> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse >>> Relative to my limited point: I do not know the status of >>> mutually-exclusive categorizations vs. not for ZFS ARC and >>> Mem. >>> >>> Unfortunately, as I understand things, it is questionable if >>> adding -S to the top command gives you swap information that >>> can point to what makes up the 15G swapped out by totaling >>> the sizes listed. But you might at least be able to infer >>> what processes became swapped out even if you can not get >>> a good size for the swap space use for each. >>> >>> Using -ores does seem like it puts the top users of resident >>> memory at the top of top's process list. >>> >>> Sufficient Active RAM use by processes that stay active will >>> tend to cause inactive processes to be swapped out. FreeBSD >>> does not swap out processes that stay active: it pages those >>> as needed instead of swapping. >>> >>> So using top -Sores might allow watching what active >>> process(es) grow and stay active and what inactive processes >>> are swapped out at the time of the activity. >>> >>> I do infer that the 15G Used for Swap is tied to processes >>> that were not active when swapped out. >>> >>>> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", >>>> >>>> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. >>>> >>>> >>>> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? >>>> >>>> Thanks. >>>> >>>> >>>> On 10/24/18 9:34 AM, Mark Millard wrote: >>>>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: >>>>> >>>>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>>>> Robert wrote: >>>>>> >>>>>>> So the issue is still happening. Please check attached screenshot. >>>>>>> The green area is "inactive + cached + free". >>>>>>> >>>>>>> . . . >>>>>> +1 >>>>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free >>>>>> Swap: 112G Total, 19M Used, 112G Free >>>>> Just a limited point based on my understanding of "Buf" in >>>>> top's display . . . >>>>> >>>>> If "cached" means "Buf" in top's output, my understanding of Buf >>>>> is that it is not a distinct memory area. Instead it totals the >>>>> buffer space that is spread across multiple states: Active, >>>>> Inactive, Laundry, and possibly Wired(?). >>>>> >>>>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. >>>>> If Buf is added to that then there is double counting of >>>>> everything included in Buf and the total will be larger >>>>> than the TotalMemory. >>>>> >>>>> Also Inact+Buf+Free may double count some of the Inact space, >>>>> the space that happens to be inactive buffer space. >>>>> >>>>> I may be wrong, but that is my understanding. >>>>> From owner-freebsd-hackers@freebsd.org Sat Oct 27 03:42:40 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58E1910D5E74 for ; Sat, 27 Oct 2018 03:42:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-22.consmr.mail.ne1.yahoo.com (sonic306-22.consmr.mail.ne1.yahoo.com [66.163.189.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E4A397A8A4 for ; Sat, 27 Oct 2018 03:42:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: zwJnZtgVM1mOGeludfG9oWedVFYUTG_HiFweCrfJxeEb_8qlDSCiXVZSxzANXXT Ao9YIfs2.9b97JIjVPCkK.d0XWBdtD039da.gNkHUkYPM3b8xVbxpQM9ZcUajWnGRYJ7WwgpNB4B i97vo25sPnxDkF5szOHq51ZlBk3DkOnxUV8_2Xo.3KeSkOqlb5oPpZaYVFWb3zxfIeouEdBzwcec 1xMUq.znqLql9URk_mWl2s9IUSKTX.Hwa.OOn8VNM9ApBcLvezcj68a8yKaLjrfIpggMXxqySDiz HblhdPlUq8JOHY_nJddx.5uO3I_d4bJNpczwFb03JAEE7qzzu9oTHB2bmxXSkwXDvy2bOSCanEAa DSZs0MMVQI7.Si49kowURiqcW_.TLVfKXLmSqqPgU2rsFBJ5EOQNE0Yt7TynURSySEEtoqiGYDNN yNIrEFJFeT3PnaNmVNzJcXq512A6ev76bZ.liIPeng7MFDdQAfcYckrIDwywSG071X4BSr9_JXCP 2.Uzbma2QtyQQf3eXH6cwTKrITAe.JymyodWzRYscEdFUa.1mRQYUPVVkLp0zKTY9qI2ASN028Ef 06Tgxm1rV1hE2iEk..YN6Q7PB.iLiAj5hGNq7grRkgfnOvn0Z7Zl8hAI78RYXhMo.L9afhLLh4ZC qGbJsW8MtByle.I8ZTn87fq4VUGETdHqcqbFavgSCDEb2SvCG7WCl2i_VeHGXKlUG5QhXKsfQoc_ Xo1lEscWwVGL_QOVM_PY_rrf7CO86PF2j5vLqP3vn2cnzumDvK9qnUeUuHqXXh.cBuwt4gX5PaNc IjK6JtAR8ilsc1yWk4PkiqQ0ZJiPz81zEvKWDtMdXjBUffQM45I__7NAvZABuqNg6n.E7O9Resge O9ViEqj45fSmG88KgUwysiIK2XZPAFH9.kjC1ButvrCBgSnpnGhvR_1fyJ6CG5avKeeQiZbXhnuK 6eO3ZVQ3t2Fyig83MSansUyxuvOQ3xhMzA1VLKNzBWnIZw69pRbC605EXfOJXKkMBpj8PyrYGbu7 ShnoCJwdsMmAm0ifFHXdSYjLBnt4QURy0BeYJxuZa0.Y- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.ne1.yahoo.com with HTTP; Sat, 27 Oct 2018 03:42:33 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp420.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 3e151b3eb4af58016b2110c5cbe634ff; Sat, 27 Oct 2018 03:42:28 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: head -r339076 amd64 -> armv7 port cross build attempt with native tools involved: hangs between a cc (wait) and its child ld (uwait) Message-Id: <33C58480-1E76-4748-83B4-CB39FAD8584A@yahoo.com> Date: Fri, 26 Oct 2018 20:42:27 -0700 To: FreeBSD Toolchain , freeBSD X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 03:42:40 -0000 In trying to amd64 -> armv7 cross build ports via poudriere-devel use with native cross tools involved (and UFS, not ZFS), I'm getting about 117 ports that built and then one that ends up stuck in wait/uwait . ^C to poudriere and restarting it repeats the stuck behavior at the same point (a cc and its ld), for example: [00:02:51] [01] [00:00:00] Building print/texinfo | texinfo-6.5,1 ps output extraction (blank lines added for each of scanning): UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND . . . 0 42312 32181 0 52 0 12904 3904 select I 1 0:00.02 sh: = poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh) 0 42974 42312 0 52 0 12904 3900 wait I 1 0:00.00 sh: = poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh) 0 42975 42974 0 52 0 10408 1840 wait IJ 1 0:00.01 = /usr/bin/make -C /usr/ports/print/texinfo configure 0 43077 42975 0 52 0 10252 1792 wait IJ 1 0:00.00 /bin/sh = -e -c (cd /wrkdirs/usr/ports/print/texinfo/work/texinfo-6.5 && = _LATE_CONFIGURE_ARGS=3D"" ; if [ -z "" ] && ./configure --help=20 0 43375 43077 0 52 0 11164 2392 wait IJ 1 0:00.19 /bin/sh = ./configure --enable-nls --prefix=3D/usr/local --localstatedir=3D/var = --mandir=3D/usr/local/man --disable-silent-rules --infodir=3D/usr 0 46850 43375 0 52 0 11164 2388 wait IJ 1 0:00.00 /bin/sh = ./configure --enable-nls --prefix=3D/usr/local --localstatedir=3D/var = --mandir=3D/usr/local/man --disable-silent-rules --infodir=3D/usr 0 46857 46850 0 52 0 11080 2060 wait IJ 1 0:00.04 /bin/sh = ./configure --disable-option-checking --prefix=3D/usr/local --enable-nls = --localstatedir=3D/var --mandir=3D/usr/local/man --disable-s 0 47796 46857 0 52 0 113840 26184 wait IJ 1 0:00.15 = /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing -mcpu=3Dcortex-a= 0 47801 47796 0 52 0 285300 39672 uwait IJ 1 0:00.22 = qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr = -dynamic-linker /libexec/ld-elf.so.1 --hash-style=3Dboth --enable-new- So the "/usr/local/bin/qemu-arm-static /usr/bin/cc . . ." creates the child "qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld . = . ." process and the two get hung up. Letting it sit for long periods does not let it progress. The full commands are (note the "-pipe" vs. the = "/tmp/conftest-6c0832.o"): /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG conftest.c and: qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr = -dynamic-linker /libexec/ld-elf.so.1 --hash-style=3Dboth = --enable-new-dtags -o conftest /usr/lib/crt1.o /usr/lib/crti.o = /usr/lib/crtbegin.o -L/usr/lib /tmp/conftest-6c0832.o -lgcc --as-needed = -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed = /usr/lib/crtend.o /usr/lib/crtn.o For reference for /tmp/conftest-6c0832.o : # ls -lTd = /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0= 832.o -rw-r--r-- 1 root wheel 4204 Oct 26 17:33:13 2018 = /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0= 832.o (I'm not using tmpfs or the like at all.) The context is based on head -r339076 an is on a Ryzen Threadripper 1950X system, natively booted (not Hyper-V). (I've not tried under Hyper-V yet.) Note: I have built ports similarly before --but the last time was back in March-May sometime. # poudriere jail -jFBSDFSSDjailArmV7 -i Jail name: FBSDFSSDjailArmV7 Jail version: 12.0-ALPHA8 Jail arch: arm.armv7 Jail method: null Jail mount: /usr/obj/DESTDIRs/clang-armv7-installworld-poud Jail fs: =20 Jail updated: 2018-10-26 16:42:55 Tree name: default Tree method: null Status: parallel_build: Building started: 2018-10-26 17:29:36 Elapsed time: 02:47:50 Packages built: 0 Packages failed: 0 Packages ignored: 0 Packages skipped: 0 Packages total: 84 Packages left: 84 # poudriere ports -l PORTSTREE METHOD TIMESTAMP PATH default null 2017-08-14 21:07:05 /usr/ports I have yet to think of a way to look into this or to work around it. But my long running build on an Orange Pi Plus 2nd Edition has finished so I'll update from that for now. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sat Oct 27 04:38:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34D1410D817B for ; Sat, 27 Oct 2018 04:38:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9BB027CA7B; Sat, 27 Oct 2018 04:38:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w9R4cKxm053390 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Oct 2018 07:38:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w9R4cKxm053390 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w9R4cJkS053389; Sat, 27 Oct 2018 07:38:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 27 Oct 2018 07:38:19 +0300 From: Konstantin Belousov To: Robert Cc: Mark Millard , FreeBSD , Mark Johnston , Rozhuk Ivan Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <20181027043819.GX5335@kib.kiev.ua> References: <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 04:38:32 -0000 On Fri, Oct 26, 2018 at 03:07:07PM -0700, Robert wrote: > Sorry, let me be more specific. > > Please look into: > https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing > (wait until charts fully loaded). > > These are all memory states and mmap\munmap stats collected. Y axis is > in MBs, X is a timeline. > > It's not a problem to understand which process produces allocations and > is being swapped. I know this for sure. > > The issue is: I strongly believe that by some reason FreeBSD kernel > fails to reuse deallocated memory properly. > > Looking into graphs we can see following: > > 1. When process allocates memory (mmap), "Active" memory increases, > "Free" memory decreases (that's expected). No, mmap(2) call only causes some small amount of the kernel memory allocation, to track the mmaped region. Pages are allocated on the first touch, lets ignore the prefaulting mechanism which is not too relevant for the discussion. Mapped page can be either active or inactive, this is determined by the usage history. > > 2. When process deallocates memory (munmap), "Inactive" memory > increases, "Active" memory decreases. Again no. Unmapped pages does not get active references from userspace which touch them, so eventually pagedaemon moves active unmapped memory to inactive. > > Memory never returns into "Free" state. That's kind of expected as well. When the last mapping of the anonymous swap-backed page is destroyed, there is no point in retaining the page content, so it is freed. For the pages backed by the file, there is no point to drop content which we already paid for by read. > > 3. At some point, when sum of "Active" and "Inactive" memory exceeds > some upper memory limits, > > OS starts to push "Inactive" memory into "Laundry" and "Swap". This > happens very quick and unexpectedly. So this is the pattern of behaviour of your applications. > > Now why OS doesn't reuse huge amounts of "Inactive" memory when calling > mmap? Because typically active and inactive pages carry some useful content, which cannot be lost. Before reusing the page, pagedaemon must ensure that the page is written to file for file-backed mappings, or to swap for anonymous mappings, so its data can be restored when needed again. If the page content is synced with the permanent storage, it is called clean, otherwise dirty. Reuse of dirty page requires changing its state to clean by write-out. Laundry is the queue where the dirty pages sit waiting for write, this is done to not clog inactive queue since the page was already processed by the page daemon and decided for laundry. > > Or my assumption about availability of "Inactive" memory is wrong? Which > one is free for allocations then? Yes, you assumptions are wrong. Active/inactive represent the usage/reference history, not the reusability (clean/dirty) state. > > Thanks. > > > On 10/24/18 11:58 AM, Mark Millard wrote: > > On 2018-Oct-24, at 1:25 PM, Robert wrote: > > > >> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: > > No screen shot made it through the list back out to those that > > get messages from the freebsd-hackers at freebsd.org reference > > in the CC. The list limits itself to text as I understand. > > > >> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free > >> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other > >> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio > >> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse > > Relative to my limited point: I do not know the status of > > mutually-exclusive categorizations vs. not for ZFS ARC and > > Mem. > > > > Unfortunately, as I understand things, it is questionable if > > adding -S to the top command gives you swap information that > > can point to what makes up the 15G swapped out by totaling > > the sizes listed. But you might at least be able to infer > > what processes became swapped out even if you can not get > > a good size for the swap space use for each. > > > > Using -ores does seem like it puts the top users of resident > > memory at the top of top's process list. > > > > Sufficient Active RAM use by processes that stay active will > > tend to cause inactive processes to be swapped out. FreeBSD > > does not swap out processes that stay active: it pages those > > as needed instead of swapping. > > > > So using top -Sores might allow watching what active > > process(es) grow and stay active and what inactive processes > > are swapped out at the time of the activity. > > > > I do infer that the 15G Used for Swap is tied to processes > > that were not active when swapped out. > > > >> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", > >> > >> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. > >> > >> > >> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? > >> > >> Thanks. > >> > >> > >> On 10/24/18 9:34 AM, Mark Millard wrote: > >>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan wrote: > >>> > >>>> On Wed, 24 Oct 2018 10:19:20 -0700 > >>>> Robert wrote: > >>>> > >>>>> So the issue is still happening. Please check attached screenshot. > >>>>> The green area is "inactive + cached + free". > >>>>> > >>>>> . . . > >>>> +1 > >>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free > >>>> Swap: 112G Total, 19M Used, 112G Free > >>> Just a limited point based on my understanding of "Buf" in > >>> top's display . . . > >>> > >>> If "cached" means "Buf" in top's output, my understanding of Buf > >>> is that it is not a distinct memory area. Instead it totals the > >>> buffer space that is spread across multiple states: Active, > >>> Inactive, Laundry, and possibly Wired(?). > >>> > >>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. > >>> If Buf is added to that then there is double counting of > >>> everything included in Buf and the total will be larger > >>> than the TotalMemory. > >>> > >>> Also Inact+Buf+Free may double count some of the Inact space, > >>> the space that happens to be inactive buffer space. > >>> > >>> I may be wrong, but that is my understanding. > >>> > > > > === > > Mark Millard > > marklmi at yahoo.com > > ( dsl-only.net went > > away in early 2018-Mar) > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Sat Oct 27 05:45:52 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72DBB10DA474 for ; Sat, 27 Oct 2018 05:45:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-30.consmr.mail.ne1.yahoo.com (sonic301-30.consmr.mail.ne1.yahoo.com [66.163.184.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 03A6B7F1CA for ; Sat, 27 Oct 2018 05:45:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: eAgNwj0VM1lV.s3oma3eYiRCYQUgZBrXptdJggPrgrWrtxYriJkJ.qE9V1GVUTF 6yFJdp1Sm01EvcWJagoLQxvkBgFyAxr5vp0coTSkrU8esC7lDIR.LmyC1qS7oFzJ838nll0ieSAh 93tOlFVYm9x6J_56n6Bh7JgzRqA6SLZUQ8cRmucnp1aHe4ZLjD2QqQFp7Ecv2ytUet56oRKCjPW0 BaJ94utXJZVnuZfy5wLBu_Z9DKXjbyYaU1fG9Tq4bM_1Myf3xm4a2w9tRKl8X2EbdR1_.Vn0vaDd wWhCC4B3.wwpImNe8bxCbwYfVrYEUPQOMK5YAOTR6nQyT3Dk3BGGSaoSXXKMYw69SaOPhOnRz3un xDr1nizhRZE99Jxu4XnLgIrksePXEg3F2VqxDL2cF1y8YzVZlxqE5g6BZydLPAULeMxrZa6f_Ukx OvkTlSirfpsF9VNwlUkuSdioaPOQaQmOuyqPupJm7WEDnZDKzaWSBIt8ftYLCYQf_LgG7S6Rk5vb yjqz6w5y7zi1LbNDxJrfFoqc.svmkjm97ppfJwmvKmkBzGFOgm4jxuQBamY6b0JNXBVsnge_NlT7 4S40MR7.Ot0MohBYRzgN__us1mDeLAbYp.iBVt5Orz963rqHj.7CtLPDjp21JHqRSvOLgcy0SYyy xZ3kJd.XshDFErueOsSKvResL34b58N9IiRRJaqBs0KsylQnGQGoysb8wTy3.nTvfu6_A.dZ6dQd JlBcuueul7kJfIdycOz.5zrGuGMjyLjJgOvl5xgu0ITTOkqU1cVuh4uxYodDVwcBl3XgDswk9DER 0FBNP1qaooGSDnYV9H9dH.mHrSoJSOhIBGP7q2hodOSY3gEVWatPEPM1Ifx33jzPYtcqgbYUpDae 5U0lar7peSOoltCCtEFdFEYDYsFScF3D1HeQFk9J2z5mQWNrnhaWEvyFwUznHuOzdbo9sW750K.b S_X6D.BdsvdeIesgxLt2_xhRsuESYrC9_7XuzcUHCVsL2SexYIKg99uT.sWKMw2CSiv6iIaXwa1w 63LGTLIFLyW2taW23ulzZQxb6Ts02gX35Z1Q9fAnykbCPkOkZ Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.ne1.yahoo.com with HTTP; Sat, 27 Oct 2018 05:45:44 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp405.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 9d2efef407054cbd30f5dde6666a65ef; Sat, 27 Oct 2018 05:35:35 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: head -r339076 amd64 -> armv7 port cross build attempt with native tools involved: hangs between a cc (wait) and its child ld (uwait) Date: Fri, 26 Oct 2018 22:35:33 -0700 References: <33C58480-1E76-4748-83B4-CB39FAD8584A@yahoo.com> To: FreeBSD Toolchain , freeBSD In-Reply-To: <33C58480-1E76-4748-83B4-CB39FAD8584A@yahoo.com> Message-Id: <78DBB85A-44C5-42CF-88F2-79E75E93CF33@yahoo.com> X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 05:45:52 -0000 [Attaching to the ld process with gdb and detaching let things continue.] On 2018-Oct-26, at 8:42 PM, Mark Millard wrote: > In trying to amd64 -> armv7 cross build ports via poudriere-devel > use with native cross tools involved (and UFS, not ZFS), I'm > getting about 117 ports that built and then one that ends up stuck > in wait/uwait . ^C to poudriere and restarting it repeats the > stuck behavior at the same point (a cc and its ld), for example: >=20 >=20 > [00:02:51] [01] [00:00:00] Building print/texinfo | texinfo-6.5,1 >=20 > ps output extraction (blank lines added for each of > scanning): >=20 > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME = COMMAND > . . . > 0 42312 32181 0 52 0 12904 3904 select I 1 0:00.02 sh: = poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh) > 0 42974 42312 0 52 0 12904 3900 wait I 1 0:00.00 sh: = poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh) > 0 42975 42974 0 52 0 10408 1840 wait IJ 1 0:00.01 = /usr/bin/make -C /usr/ports/print/texinfo configure > 0 43077 42975 0 52 0 10252 1792 wait IJ 1 0:00.00 /bin/sh = -e -c (cd /wrkdirs/usr/ports/print/texinfo/work/texinfo-6.5 && = _LATE_CONFIGURE_ARGS=3D"" ; if [ -z "" ] && ./configure --help=20 > 0 43375 43077 0 52 0 11164 2392 wait IJ 1 0:00.19 /bin/sh = ./configure --enable-nls --prefix=3D/usr/local --localstatedir=3D/var = --mandir=3D/usr/local/man --disable-silent-rules --infodir=3D/usr > 0 46850 43375 0 52 0 11164 2388 wait IJ 1 0:00.00 /bin/sh = ./configure --enable-nls --prefix=3D/usr/local --localstatedir=3D/var = --mandir=3D/usr/local/man --disable-silent-rules --infodir=3D/usr > 0 46857 46850 0 52 0 11080 2060 wait IJ 1 0:00.04 /bin/sh = ./configure --disable-option-checking --prefix=3D/usr/local --enable-nls = --localstatedir=3D/var --mandir=3D/usr/local/man --disable-s >=20 > 0 47796 46857 0 52 0 113840 26184 wait IJ 1 0:00.15 = /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing -mcpu=3Dcortex-a= >=20 > 0 47801 47796 0 52 0 285300 39672 uwait IJ 1 0:00.22 = qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr = -dynamic-linker /libexec/ld-elf.so.1 --hash-style=3Dboth --enable-new- >=20 > So the "/usr/local/bin/qemu-arm-static /usr/bin/cc . . ." > creates the child "qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld = . . ." > process and the two get hung up. Letting it sit for long periods does > not let it progress. >=20 > The full commands are (note the "-pipe" vs. the = "/tmp/conftest-6c0832.o"): >=20 > /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing = -mcpu=3Dcortex-a7 -DLIBICONV_PLUG conftest.c >=20 > and: >=20 > qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr = -dynamic-linker /libexec/ld-elf.so.1 --hash-style=3Dboth = --enable-new-dtags -o conftest /usr/lib/crt1.o /usr/lib/crti.o = /usr/lib/crtbegin.o -L/usr/lib /tmp/conftest-6c0832.o -lgcc --as-needed = -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed = /usr/lib/crtend.o /usr/lib/crtn.o >=20 > For reference for /tmp/conftest-6c0832.o : >=20 > # ls -lTd = /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0= 832.o > -rw-r--r-- 1 root wheel 4204 Oct 26 17:33:13 2018 = /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0= 832.o >=20 > (I'm not using tmpfs or the like at all.) >=20 >=20 > The context is based on head -r339076 an is on a > Ryzen Threadripper 1950X system, natively booted > (not Hyper-V). (I've not tried under Hyper-V yet.) >=20 > Note: I have built ports similarly before --but the > last time was back in March-May sometime. >=20 > # poudriere jail -jFBSDFSSDjailArmV7 -i > Jail name: FBSDFSSDjailArmV7 > Jail version: 12.0-ALPHA8 > Jail arch: arm.armv7 > Jail method: null > Jail mount: /usr/obj/DESTDIRs/clang-armv7-installworld-poud > Jail fs: =20 > Jail updated: 2018-10-26 16:42:55 > Tree name: default > Tree method: null > Status: parallel_build: > Building started: 2018-10-26 17:29:36 > Elapsed time: 02:47:50 > Packages built: 0 > Packages failed: 0 > Packages ignored: 0 > Packages skipped: 0 > Packages total: 84 > Packages left: 84 >=20 > # poudriere ports -l > PORTSTREE METHOD TIMESTAMP PATH > default null 2017-08-14 21:07:05 /usr/ports >=20 >=20 > I have yet to think of a way to look into this or to work around > it. But my long running build on an Orange Pi Plus 2nd Edition > has finished so I'll update from that for now. I tried again and when it hung up I used gdb to attach to the ld process and later to detach: # gdb `which qemu-arm-static` . . . (gdb) attach 18703 Attaching to program: /usr/local/bin/qemu-arm-static, process 18703 Couldn't get registers: Device busy. . . . (gdb) bt #0 _umtx_op () at _umtx_op.S:3 #1 0x0000000060050cd4 in _umtx_wait_uint_private (where=3D0x0, = addr=3D, target_val=3D, tsz=3D, t=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/freebsd/os-thread.c:258 #2 freebsd_lock_umutex (target_addr=3D4102556064, id=3D100867, ts=3D0x0, = mode=3D) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/freebsd/os-thread.c:890 #3 0x000000006004a808 in do_freebsd__umtx_op (obj=3D4102556064, = op=3D, val=3D0, uaddr=3D0, target_time=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/freebsd/os-thread.h:359 #4 0x00000000600414d5 in do_freebsd_syscall (cpu_env=3D0x8607a4c58, = num=3D454, arg1=3D, arg2=3D, = arg3=3D, arg4=3D0, arg5=3D0, arg6=3D-185272152, arg7=3D0, = arg8=3D0) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/syscall.c:1364 #5 0x0000000060038d03 in target_cpu_loop (env=3D0x8607a4c58) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/arm/target_arch_cpu.h:207 #6 0x00000000600386a9 in cpu_loop (env=3D0xf48809bc) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/main.c:121 #7 0x0000000060039922 in main (argc=3D-10608, argv=3D0x7fffffffd1d8) at = /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/b= sd-user/main.c:513 (gdb) detach Detaching from program: /usr/local/bin/qemu-arm-static, process 18703 Things started back up from there. We will see if it hangs up again. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sat Oct 27 06:52:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25E4B10DC5C1 for ; Sat, 27 Oct 2018 06:52:13 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D5B181312 for ; Sat, 27 Oct 2018 06:52:12 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-ot1-x342.google.com with SMTP id c32so3102234otb.8 for ; Fri, 26 Oct 2018 23:52:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/zgqvBk1+KTF0wNV/9rzEB09yWDQML/lLvPOZRQFf9M=; b=Jbo/PDf/TAAvQwpmVoc63g4Ch9d6QNFEUE6FxOkYGJVDimcfxxhyTH/WKgcXOJlUBe s/TAgIRMz4jOmjTae2gVqAWzhgmKgVr2faIfFnyUj4bkFx7+pj/KlJZdeBbKFDCojiYb xCOtls3rVlEw/4U1RPc7vQww0e/UH9N66PVKCBq9JcLkuXE425svvqEBMg8xFMkDXHpa Rc5S2upzHg+Px/D7ahoTA7JdaGTxLyHcOrZ5USn8792o0FS42/Tp4BihVMej/FwFrtZP lj+HwBXKfHHqH11jiuA6lLodC4Y25TmjFUeCgw/c8rGFPzaufTfoO6YMccNWthlqplXi IJbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/zgqvBk1+KTF0wNV/9rzEB09yWDQML/lLvPOZRQFf9M=; b=bimq2suD86ndqTuwBKVH30lZad0Ji/1J6L1KBqsmnqI5P+Lp9fzyOTq2IqZsDtITBw v9uaRgVDoOX1Qvq2PjJFWOpewSIIS1u9tTiNObzHMPHIrv0wAjfZS0fqqgF2bmhF53wy YQqB9WN1lpIFmqRjrZx+EP66BtnRUseGWb8pl78fwOxb0ctREBlscnnI8Xz9c3zYudds y7pn/O9K66Zq3jnOAUvl74ho+i65bbs+AIp2NyPTVsKHhYrCou53tB8s2rLTLnfLMzp4 srVn1MmSVrK3uHJhg8Zjq2a8GV0iDBVWMeQ4V5aptLYiEmBDHsoywYwsuKIR47q+gMB1 DnvA== X-Gm-Message-State: AGRZ1gI0mJakv9i35RJOoJUZyYNr59JCRirLvh0rgXmTuBsIukIPteAB DrzHAn/HE2ypMP2yMks84umqBMTCLfTYTcDUOeQ= X-Google-Smtp-Source: AJdET5fhkHYwjymL2F0SYNwQdO5//UFTeXGZVe+zowWhp4GeGx+nxm+w8SAFhDyoBipq32MsHWihH4lkYFx9o3Sqck4= X-Received: by 2002:a9d:420b:: with SMTP id q11mr3847530ote.130.1540623131762; Fri, 26 Oct 2018 23:52:11 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Steevan Rodrigues Date: Sat, 27 Oct 2018 12:22:00 +0530 Message-ID: Subject: Re: Contigfree takes too much time that cuases PCIe driver unload to take up to 30 minutes To: rysto32@gmail.com Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 06:52:13 -0000 Yes , it is contigfree that is taking too much time. ( Contigfree is taking almost 5x to 10x more time than contigmalloc ) However, I see this issue only on a SuperMicro Server which has Xeon Gold CPU with 10 cores ( dual CPU i.e total 20 cores). So I am wondering whether FreeBSD has performance issues on mutlicore ( more than 16 cores) servers ? Has anyone come across issues like this? Thanks Steevan On Fri, Oct 5, 2018 at 11:31 PM Ryan Stone wrote: > I'm not sure why configfree() would be taking a long time, but > configmalloc() can be extraordinarily expensive when it needs to > defragment memory to meet the request. Does your application really > require a lot of physically contiguous memory? if you can restructure > to not require contigmalloc() all -- maybe by using S/G DMA -- you may > find your life significantly easier. > From owner-freebsd-hackers@freebsd.org Sat Oct 27 07:46:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B566410DE0E4 for ; Sat, 27 Oct 2018 07:46:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-15.consmr.mail.bf2.yahoo.com (sonic315-15.consmr.mail.bf2.yahoo.com [74.6.134.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D5FE8336E for ; Sat, 27 Oct 2018 07:46:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Zz9G2QgVM1lUlGB_oVBVLyJLdoHJJ8FQ.hjIL_JvvSWRAdmitPRKOrzR99oOzlQ 5TbIMWsU4cLnkoXCsrRCrJS2anhY_J7hD.1515qR_cMyGqI382Lbl4RZ3ZmahP7z8RR0UZY2yFuU KUYCDhAVuS8xyFY9FItFHnQ3BRAoAzUSqzuIIHS0MGFjQV2AJWIq4Ha3ohwLqgo9_mZWtBXRuaFZ LixxwvswJ4HSFU6A0iINe2M1FaXuBryTd79IeDd9KReK2fdnTCsW1mmVGkLUZ0JgOM3lnRekUhp7 rphU4TBYu3eUFHgabPopAICyS5THveHgQktSLT2y7Tcmw6onfZuaDv.s6R1RCdFFuluWEWoBI0Lw fFLexNguNn_9FKHaYsoqVS6RRbhyquV0Dzf_Wnx8_E2Sykfqt_DjBtNBNM2enxP8mb_CzV91qyfn FVZLx95yjyye84gTTilaGPt8MQTs7EySqTPKKvvegihHltZqpDUFSYVk4P0ZXkN_eDoYdJEib0MK T9JVboOineEkhFDHP3ykfSpCmJtm27mKiC.t8ACg3xT3SmcKnuPo7Ugv5omncx6YgtTq9oLGiatH y4PWnb6i73.XJHlGLbL40TA7eph9O0AAebFFw_irrAdASfGPbmcPjFkyzXaArJqOKe.KbdLR2Rel lW1cWOgjfrQEPrk0fcD2FR.aNWnOXcZR8.IJ7AYLTRSnMe9kp.d0BLrBgylzMrjJOO8HSsFcgQnM JmHPmWwz21XgQ.ds8WxdE0MeMeK9_GsIwGdvNbhEsrVBqwnqwmGemcdDeOHMZx54hfGw754zWluj aKgE_A1K89zZtZD_fUy259gQ928_ciGtAJawKWzevGr84ygDQsLQTvULvo31EYgzjxZAFYr6LnR1 RU2cfSmC09X_SLN.a1KrdmSCaz9xUUIfyvaz3JJDDzJFTIuCnlBqBQ7DNoPLTccf.r2yFu._.HW1 rlWm3yNucUVosTvUAC9YE6nBWt39NTOFb9om2_i3_zJkX9z7nYbxb7C.OkvLoprNKwRbqseRB0Aq LCPZNDgaCspJEtyZQr.GfDiwwwPYy9wYh23L1O2pZu3cR Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.bf2.yahoo.com with HTTP; Sat, 27 Oct 2018 07:45:54 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp404.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID b06008946d478eccb87023b5ac10de65; Sat, 27 Oct 2018 07:45:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Sudden grow of memory in "Laundry" state From: Mark Millard In-Reply-To: Date: Sat, 27 Oct 2018 00:45:48 -0700 Cc: Rozhuk Ivan , FreeBSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> To: Robert X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 07:46:01 -0000 On 2018-Oct-26, at 10:36 PM, Robert = wrote: > Hi Mark, thanks for you reply.=20 > Regarding memory flags - I'm not setting them directly anywhere. >=20 > Initially my app allocates shared memory segment of size RAM \ 2: >=20 > namespace bip =3D boost::interprocess; > bip::managed_shared_memory(bip::open_or_create, SHMEM_SEG_NAME > , size) >=20 > and never resizes\deallocates it. >=20 >=20 > Later it uses only "new"-s and "mallocs". Ahh. I was thinking that you were using mmap and munmap directly. new/delete and malloc/free may well manage pool of memory areas and not allocate or un-allocate at the OS level 1-to-1 with the new/malloc or delete/free calls. There is a whole extra layer of policy/management involved. There are likely far fewer mmaps than new's+malloc's and far fewer munmap's than delete's+free's. The documentation indicates that the destructor for bip::managed_shared_memory does an unmap of some kind and frees resources --but does not remove the named object from the system. Looking around this appears to mean that a backing-store file is left around and reused (for the same memory segment name). For FreeBSD it does appear to use mmap/munmap and the default_map_options lead to MAP_NOSYNC use. I saw no evidence of MADV_FREE use. But there is code involving MADV_DONTNEED for UNIXes that do not have destructive semantics for it, FreeBSD being indicated as one as I remember. > When I watched for flags in DTRACE, they were all like below: >=20 > 2018 Oct 20 11:24:48 mmap args: addr:0 len:2097152 prot:3 flags:1002 = fd:-1 offset:0, which is: >=20 > # > define MAP_PRIVATE 0x0002 /* changes are private */ > and > # > define MAP_ANON 0x1000 /* allocated from memory, swap = space=20 > */ >=20 >=20 > This is what alloc\new generates by default. I see. > Btw, if you look into mmap\munmap sizes you will notice it generates = 3-5 GB more mmaps than munmaps every 5 minutes. > This means machine should be out of memory within an hour. Fragmentation of managed areas such that having even one part of an area in use means that the area can not be given back to the OS? > But it doesn't happen so fast... > Why DTRACE lies here? Or is it because DTRACE can't catch all munmaps = due to optimizations in kernel code, which was discussed recently in = mailing list? I've no evidence that DTRACE is wrong here: there may have been new's = without matching delete's or malloc's without matching free's at the time. In = such cases there would be fewer unmaps than maps at the time in order to = preserve what is still allocated. Of course, I've no evidence for the other direction either. > Anyway, from your response I understood that using MADV_FREE may help.=20= > Any idea of how to use it properly? Should I call madvise after every = free\delete on the "freed" pages? This sounds like something completely = wrong... Normal new/delete and malloc/free use would not use MADV_FREE: explicit page management is not visible at those programming interfaces. It appears that I have way to little context to be of much help. > On 10/26/18 5:40 PM, Mark Millard wrote: >> On 2018-Oct-26, at 3:07 PM, Robert = wrote: >>=20 >>=20 >>> Sorry, let me be more specific. >>>=20 >>> Please look into:=20 >>> = https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4Khcvd= pOL68gEUM/edit?usp=3Dsharing >>> (wait until charts fully loaded). >>>=20 >> Thanks for giving folks access to the charts originally referred to. >>=20 >>=20 >>> These are all memory states and mmap\munmap stats collected. Y axis = is in MBs, X is a timeline. >>>=20 >> Some things folks looking into this might want to know: >>=20 >> MAP_PRIVATE in use? Vs.: MAP_SHARED in use? >>=20 >> MAP_NOSYNC in use or not? >>=20 >> MAP_ANON in use or not? >>=20 >> MAP_FIXED in use or not? (Probably not?) >>=20 >> But I cover MAP_NOSYNC and another option that is >> in a 3rd call below and do not need such information >> for what I've written. >>=20 >>=20 >>> It's not a problem to understand which process produces allocations = and is being swapped. I know this for sure. >>>=20 >>> The issue is: I strongly believe that by some reason FreeBSD kernel = fails to reuse deallocated memory properly. >>>=20 >>> Looking into graphs we can see following: >>>=20 >>> 1. When process allocates memory (mmap), "Active" memory increases, = "Free" memory decreases (that's expected). >>>=20 >>> 2. When process deallocates memory (munmap), "Inactive" memory = increases, "Active" memory decreases. >>>=20 >>> Memory never returns into "Free" state. That's kind of expected as = well. >>>=20 >> =46rom the description of MAP_NOSYNC for mmap >> (vs. the flushing behavior): >>=20 >> . . . Without this option any VM >> pages you dirty may be flushed to disk every so = often >> (every 30-60 seconds usually) which can create = perfor- >> mance problems if you do not need that to occur = (such >> as when you are using shared file-backed mmap = regions >> for IPC purposes). Dirty data will be flushed = auto- >> matically when all mappings of an object are = removed >> and all descriptors referencing the object are = closed. >> Note that VM/file system coherency is maintained >> whether you use MAP_NOSYNC or not. >>=20 >> Note the specified behavior for flushing out "dirty data" >> unless MAP_NOSYNC is in use. (I note another alternative >> later.) >>=20 >> As I understand it FreeBSD uses the swapping/paging code to do the >> flush activity: part of the swap/page space is mapped into the >> the file in question and the flushing is a form of swapping/paging >> out pages. >>=20 >> [Note: Top does not keep track of changes in swap space, >> for example a "swapon" done after top has started >> displaying things will not show an increased swap total >> but the usage can show larger than the shown total. >> Flushing out to a mapped file might be an example of >> this for all I know.] >>=20 >> Also reported for flushing behavior is: >>=20 >> . . . The fsync(2) system call will flush all dirty data and >> metadata associated with a file, including dirty >> NOSYNC VM data, to physical media. The sync(8) >> command and sync(2) system call generally do not >> flush dirty NOSYNC VM data. The msync(2) system >> call is >> usually not needed since BSD implements a = coherent >> file system buffer cache. However, it may be = used to >> associate dirty VM pages with file system = buffers and >> thus cause them to be flushed to physical media = sooner >> rather than later. >>=20 >>=20 >> As for munmap: its description is that the address range is still >> reserved afterwards, quoting the description: >>=20 >> The munmap () system call deletes the mappings and guards for the = speci- >> fied address range, and causes further references to addresses = within the >> range to generate invalid memory references. >>=20 >> That last is not equivalent to the address range being "free" >> in that the range still counts against the process address space. >> (So being accurate about what is about RAM availability vs. address >> space usage/availability is important in order to avoid confusions.) >>=20 >> It would appear that to force invalid memory references involves >> keeping page descriptions around but they would be inactive, >> rather than active. This is true no matter if RAM is still associated >> or not. (So this could potentially lead to a form of extra counting >> of RAM use, sort of like in my original note.) See later below for >> another means of control . . . >>=20 >> Remember: "Dirty data will be flushed automatically when all mappings = of >> an object are removed and all descriptors referencing the object are >> closed". So without MAP_NOSYNC the flushing is expected. But see = below >> for another means of control . . . >>=20 >> There is another call madvise that has an option tied >> to enabling freeing pages and avoiding flushing the >> pages: >>=20 >> MADV_FREE Gives the VM system the freedom to free pages, and = tells >> the system that information in the specified page = range >> is no longer important. This is an efficient way = of >> allowing malloc(3) to free pages anywhere in the = address >> space, while keeping the address space valid. The = next >> time that the page is referenced, the page might = be >> demand zeroed, or might contain the data that was = there >> before the MADV_FREE call. References made to = that >> address space range will not make the VM system = page the >> information back in from backing store until the = page is >> modified again. >>=20 >> This is a way to let the system free page ranges and >> allow later use of the address range in the process's >> address space. No descriptions of page ranges that should >> generate invalid memory references, so no need of such >> "inactive pages". >>=20 >> MADV_FREE makes clear that your expectations of the meaning >> of munmap does not seem to match FreeBSD's actual usage: >> MADV_FREE must be explicit to get the behavior you appear >> to be looking for. At least that is the way I read the >> documentation's meaning. MAP_NOSYNC does not seem sufficient >> for matching what you are looking for as the behavioral >> properties --but it appears possibly necessary up to when >> MADV_FREE can be used. >>=20 >>=20 >>> 3. At some point, when sum of "Active" and "Inactive" memory exceeds = some upper memory limits, >>>=20 >>> OS starts to push "Inactive" memory into "Laundry" and "Swap". This = happens very quick and unexpectedly. >>>=20 >> This is the flushing activity documented as far as I can tell. >>=20 >>=20 >>> Now why OS doesn't reuse huge amounts of "Inactive" memory when = calling mmap? >>>=20 >> Without MADV_FREE use the system does not have "the freedom >> to free pages". Without MAP_NOSYNC as well it is expected >> to flush out some pages at various times as things go along. >>=20 >>=20 >>> Or my assumption about availability of "Inactive" memory is wrong? = Which one is free for allocations then? >>>=20 >> Pages that are inactive and dirty, normally have to be >> flushed out before the RAM for the page can be freed >> for other uses. MADV_FREE is for indicating when this is >> not the case and the usage of the RAM has reach a stage >> where the RAM can be more directly freed (no longer tied >> to the process). >>=20 >> At least that is my understanding. >>=20 >> Mark Johnston had already written about MADV_FREE but not >> with such quoting of related documentation. If he and I >> seem to contradict each other anywhere, believe Mark J. >> I'm no FreeBSD expert. I'm just trying to reference and >> understand the documentation. >>=20 >>=20 >>> Thanks. >>>=20 >>>=20 >>> On 10/24/18 11:58 AM, Mark Millard wrote: >>>=20 >>>> On 2018-Oct-24, at 1:25 PM, Robert = wrote: >>>>=20 >>>>=20 >>>>> Sorry, that wasn't my output, mine (related to the screenshot I've = sent earlier) is: >>>>>=20 >>>> No screen shot made it through the list back out to those that >>>> get messages from the freebsd-hackers at freebsd.org reference >>>> in the CC. The list limits itself to text as I understand. >>>>=20 >>>>=20 >>>>> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M = Free >>>>> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M = Other >>>>> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio >>>>> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse >>>>>=20 >>>> Relative to my limited point: I do not know the status of >>>> mutually-exclusive categorizations vs. not for ZFS ARC and >>>> Mem. >>>>=20 >>>> Unfortunately, as I understand things, it is questionable if >>>> adding -S to the top command gives you swap information that >>>> can point to what makes up the 15G swapped out by totaling >>>> the sizes listed. But you might at least be able to infer >>>> what processes became swapped out even if you can not get >>>> a good size for the swap space use for each. >>>>=20 >>>> Using -ores does seem like it puts the top users of resident >>>> memory at the top of top's process list. >>>>=20 >>>> Sufficient Active RAM use by processes that stay active will >>>> tend to cause inactive processes to be swapped out. FreeBSD >>>> does not swap out processes that stay active: it pages those >>>> as needed instead of swapping. >>>>=20 >>>> So using top -Sores might allow watching what active >>>> process(es) grow and stay active and what inactive processes >>>> are swapped out at the time of the activity. >>>>=20 >>>> I do infer that the 15G Used for Swap is tied to processes >>>> that were not active when swapped out. >>>>=20 >>>>=20 >>>>> I'm OK with a low "Free" memory if OS can effectively allocate = from "Inactive", >>>>>=20 >>>>> but I'm worrying about a sudden move of a huge piece of memory = into "Swap" without any relevant mmap calls. >>>>>=20 >>>>>=20 >>>>> My question is: what else (except mmap) may reduce "Free" memory = and increase "Laundry"\"Swap" in the system? >>>>>=20 >>>>> Thanks. >>>>>=20 >>>>>=20 >>>>> On 10/24/18 9:34 AM, Mark Millard wrote: >>>>>=20 >>>>>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan = wrote: >>>>>>=20 >>>>>>=20 >>>>>>> On Wed, 24 Oct 2018 10:19:20 -0700 >>>>>>> Robert=20 >>>>>>> >>>>>>> wrote: >>>>>>>=20 >>>>>>>=20 >>>>>>>> So the issue is still happening. Please check attached = screenshot. >>>>>>>> The green area is "inactive + cached + free". >>>>>>>>=20 >>>>>>>> . . . >>>>>>>>=20 >>>>>>> +1 >>>>>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M = Buf, 617M Free >>>>>>> Swap: 112G Total, 19M Used, 112G Free >>>>>>>=20 >>>>>> Just a limited point based on my understanding of "Buf" in >>>>>> top's display . . . >>>>>>=20 >>>>>> If "cached" means "Buf" in top's output, my understanding of Buf >>>>>> is that it is not a distinct memory area. Instead it totals the >>>>>> buffer space that is spread across multiple states: Active, >>>>>> Inactive, Laundry, and possibly Wired(?). >>>>>>=20 >>>>>> In other words: TotalMemory =3D Active+Inact+Laundry+Wired+Free. >>>>>> If Buf is added to that then there is double counting of >>>>>> everything included in Buf and the total will be larger >>>>>> than the TotalMemory. >>>>>>=20 >>>>>> Also Inact+Buf+Free may double count some of the Inact space, >>>>>> the space that happens to be inactive buffer space. >>>>>>=20 >>>>>> I may be wrong, but that is my understanding. >>>>>=20 >>>>>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sat Oct 27 23:43:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE4BA10D857E for ; Sat, 27 Oct 2018 23:43:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-22.consmr.mail.gq1.yahoo.com (sonic312-22.consmr.mail.gq1.yahoo.com [98.137.69.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 45ECF82A75 for ; Sat, 27 Oct 2018 23:43:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Z7hhIFsVM1nCyDl0ZupAd9NoLHYkueSsCTBD0dOR8BeB7GxKVKVyr5dRBTUJwZM 3y3.QuQxUKRjcV2V6Niz2CqTIkl9ZPTpGlJb3lfnlZjrjKFf5BJm_5RZTpZIXkUsAltNz8Coy990 4XPVVGcnItgQJbEp7vMmMkgIDhhJSvC0fWNCgVayuqUCdyRBoKAmDDugGmJYapwvw5Mvm1DwkrS8 ljrFvqCS3KOwElgYe97ELvm77lFm7LgpLUz7MneW4StDmbI0OChBSonJ6hX5_6z9PZkGrvHB4URk 76DdWSHo7hvXu9.5XdA23r_dqzNUFR_NKQTDfBPz.GoBnE9evg_YjVzO8kSYVOeuxhskM7p01o5F 1CZsCgZDo_8TiB0ka1Lf1miYAYIICrmpgM7wCtD_dFtnJ_lrEOxq3XCFM1BzntjiKH7ctQUr8wqH 473tNtyvQ60XiajoRbsNzr_rNNXkt3qK.vQn9_c_1VnetRIIXTOhf8iYzGvhJ0qQE.850PDKMpn0 tYNxftekru4lQ7NsCYDWP0uTyT7zLP4.113VZiHket0kw9lmpT5tbRFDyZOXVIj8Hj9bsuTu3Y3S aeHLcdpkmSif_EcYcml06fdVJOka4zBEcTx9nVjtdH2Rcx5vIhX4MYffxK33xVTtFKykNwsc93et 519dxjW3G5N17.KZYuQYu4Fy_k2Pvfo9e8GTZnmjbTg1hJ8j.0F0Iwcqfh7w_L8sgY7oTC5Islcl 0Ik5ZYq7.u16kDLgzBjMmRB3w9WVf8QVNFjIQaZj5f00s2G2Ju10g0fRFGxz_.Bn5.jWIaKs.HqP 8nZrGCpxRkY7s_h1sgLVUDOk8MBbLc9hhNarRzpiAriKDnIwDr.GP6UH3InXUVdY5ko_iyByX2ho ozc.yqPj_Sgd0Ao7OFFkZ1GfYptio4qBVRToSS8zwvRmlru2QgjfsfSOo7cY0fGu8ijGXnR9VOvG w0cCSRFzU5Dg5viSua7leyzLkJKhBg87l9c9wFl2i2NhkQvPh4uUEazc9nzUDMUMlIGQhK1sD9hI TVvmoq7cTQrVrX6It8BRFuvyL6Md4j7JRM3jnXHeGE9eRAXeqTbrIbTY4Z9m7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Sat, 27 Oct 2018 23:43:15 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp426.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID ee0cc28ffc4714a30c4886e4775cf224; Sat, 27 Oct 2018 23:33:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: head -r339076 amd64 -> armv7 port cross build attempt with native tools involved: hangs between a cc (wait) and its child ld (uwait) From: Mark Millard In-Reply-To: Date: Sat, 27 Oct 2018 16:33:03 -0700 Cc: FreeBSD Toolchain , freeBSD , FreeBSD Ports ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <33C58480-1E76-4748-83B4-CB39FAD8584A@yahoo.com> To: =?utf-8?Q?Mika=C3=ABl_Urankar?= , Sean Bruno X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Oct 2018 23:43:24 -0000 [Some of this discussion occurred off list. The point here is not specific to the hang that I originally reported.] On 2018-Oct-27, at 3:03 PM, Mark Millard wrote: >=20 >> . . . >>=20 >>=20 >>> There are bugs in qemu that can cause such deadlock, you can try = these >>> 2 patches: >>> = https://github.com/MikaelUrankar/qemu-bsd-user/commit/9424a5ffde4de2768ab6= baa45fdbe0dbb56a7371 >>> = https://github.com/MikaelUrankar/qemu-bsd-user/commit/d6f65a7f07d280b6906d= 499d8e465d4d2026c52b >>=20 >> I'll try those later. Thanks. (I need to get back to sleep.) >>=20 >> It was interesting that attach/detach to the ld process >> caused it to progress. The rest of the build completed >> just fine. But that one spot consistently hung up before >> trying gdb to look at the back trace. >>=20 >=20 > Looking at the qemu code related to the 2nd patch: the > structure of the field copies (via __get_user) seems > very sensitive to the ABI rules for the target and > how things align and such, given that the structure > description and code are host code. __packed vs. not > is possibly not sufficient control to always make things > match right across all the potential combinations of > host and target from what I can see. >=20 > Lack of __packed may prove sufficient for my specific > context (amd64 host and armv7 target) but it seems > non-obvious what to do in general. >=20 > There would also seem to be big endian vs. little endian > issues on the individual __get_user styles of copies > when the host and target do not match for a multi-byte > numeric encoding. Well, I get the following for: #include "/usr/include/sys/event.h" // kevent #include // offsetof #include // printf int main() { printf("%lu\n", (unsigned long) sizeof(struct kevent)); printf("ident %lu\n", (unsigned long) offsetof(struct kevent, = ident)); printf("filter %lu\n", (unsigned long) offsetof(struct kevent, = filter)); printf("flags %lu\n", (unsigned long) offsetof(struct kevent, = flags)); printf("fflags %lu\n", (unsigned long) offsetof(struct kevent, = fflags)); printf("data %lu\n", (unsigned long) offsetof(struct kevent, = data)); printf("udata %lu\n", (unsigned long) offsetof(struct kevent, = udata)); printf("ext %lu\n", (unsigned long) offsetof(struct kevent, = ext)); return 0; } (This code avoided warnings for type mismatches with the printf strings and such.) amd64 native [host of qemu use] (comments hand added): # ./a.out 64 ident 0 filter 8 // NOTE! flags 10 // NOTE! fflags 12 // NOTE! data 16 udata 24 ext 32 (The above is not particularly important but I include it for completeness.) armv7 native [target in qemu use] (comments hand added): # ./a.out 64 // NOTE vs. below! ident 0 filter 4 // NOTE vs. above! flags 6 // NOTE vs. above! fflags 8 // NOTE vs. above! data 16 // NOTE vs. below! udata 24 // NOTE vs. below! ext 32 // NOTE vs. below! /usr/include/sys/event.h lacks __packed in both cases. With __packed in qemu-arm-static's source code for target_freebsd_kevent I confirm that via gdb for the qemu-arm-static: p/d sizeof(struct target_freebsd_kevent) p/d &((struct target_freebsd_kevent *)0)->ident p/d &((struct target_freebsd_kevent *)0)->filter p/d &((struct target_freebsd_kevent *)0)->flags p/d &((struct target_freebsd_kevent *)0)->fflags p/d &((struct target_freebsd_kevent *)0)->data p/d &((struct target_freebsd_kevent *)0)->udata p/d &((struct target_freebsd_kevent *)0)->ext reports as the 2nd patch's problem-report material reports (56,0,4,6,8,12,20,24): not even the right size. I also confirm that removing __packed in qemu's code and rebuilding and then checking with gdb reported a match to the above armv7 native report (64,0,4,6,8,16,24,32). I have not verified __packed used vs. not for any other combination of host and target platforms. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)