From owner-freebsd-stable@freebsd.org Mon Nov 18 13:07:09 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 912281BF599 for ; Mon, 18 Nov 2019 13:07:09 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Gq3w3235z4MhD; Mon, 18 Nov 2019 13:07:08 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAID6vRx026284 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 18 Nov 2019 13:07:00 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: delphij@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAID6nLn052686 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 18 Nov 2019 20:06:49 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: bhyve memory leak in stable/11 To: Xin LI References: <7fddcea5-2188-afe1-3ea9-a53dffdbec32@grosbein.net> Cc: Alexander Motin , FreeBSD stable From: Eugene Grosbein Message-ID: <0cb84655-bdd2-1881-cfa2-09875c0aa7ff@grosbein.net> Date: Mon, 18 Nov 2019 20:06:44 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47Gq3w3235z4MhD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.72 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.62)[ip: (-4.17), ipnet: 2a01:4f8::/29(-2.30), asn: 24940(-1.64), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Nov 2019 13:07:09 -0000 18.11.2019 19:03, Eugene Grosbein wrote: > Please point me to right direction for debugging this. Is it normal that over 1/3rd of 360G total physical RAM is in "Laundry" category in addition to 173G Wired? last pid: 20372; load averages: 8.04, 7.73, 7.84 up 2+05:55:29 16:04:02 130 processes: 3 running, 126 sleeping, 1 zombie CPU: 1.1% user, 0.0% nice, 13.2% system, 0.1% interrupt, 85.7% idle Mem: 42G Active, 8325M Inact, 112G Laundry, 173G Wired, 7809M Free ARC: 131G Total, 28G MFU, 90G MRU, 11M Anon, 2442M Header, 10G Other 107G Compressed, 363G Uncompressed, 3.41:1 Ratio Swap: 64G Total, 16G Used, 48G Free, 24% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 78042 root 34 20 0 54328M 52867M kqread 7 81.4H 210.63% bhyve: sappdev (bhyve) 59085 root 20 20 0 31512M 25256M kqread 6 490:16 16.25% bhyve: sdc01 (bhyve) 59568 root 28 20 0 28549M 24270M kqread 6 143:32 1.22% bhyve: sfile01 (bhyve) 60011 root 20 20 0 30262M 23697M kqread 27 121:22 1.08% bhyve: skms01 (bhyve) 63676 root 34 20 0 16418M 12799M kqread 3 113:06 19.92% bhyve: solap (bhyve) 26819 root 26 20 0 12321M 10472M kqread 28 151:43 10.12% bhyve: srdapp01 (bhyve) 63662 root 34 20 0 8226M 6969M kqread 4 114:52 20.36% bhyve: ssql01 (bhyve)