From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:04:33 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9AF61526680 for ; Mon, 11 Mar 2019 17:04:33 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from mail.io7m.com (mail.io7m.com [IPv6:2001:19f0:5:752:f000::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.io7m.com", Issuer "arc7 CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D19A70649 for ; Mon, 11 Mar 2019 17:04:33 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from almond.int.arc7.info (cust187-dsl61.idnet.net [212.69.61.187]) by mail.io7m.com (Postfix) with ESMTPSA id 917922F29E for ; Mon, 11 Mar 2019 17:04:30 +0000 (UTC) Date: Mon, 11 Mar 2019 17:04:20 +0000 From: Mark Raynsford To: freebsd-virtualization@freebsd.org Subject: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311170420.61bd34c6@almond.int.arc7.info> Organization: io7m.com OpenPGP: id=E379595716131DB4E7AB87DD67B18CC1F9ACE345; url=http://io7m.com/pgp/E379_5957_1613_1DB4_E7AB_87DD_67B1_8CC1_F9AC_E345.key MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/xfmX_SEqiC4xTlJL4YLkLBj"; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:04:34 -0000 --Sig_/xfmX_SEqiC4xTlJL4YLkLBj Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hello! Leaving aside userland monitoring tools such as Prometheus[0], is there any way to detect on the host that a guest kernel running in bhyve has booted? I'm assuming "booted" in this sense to mean "PID 1 has probably started". I'm guessing that there probably isn't, but I thought I'd better ask anyway. "Not booted" would mean something like "Is sitting at the Grub prompt doing nothing". [0] https://prometheus.io/ --=20 Mark Raynsford | http://www.io7m.com --Sig_/xfmX_SEqiC4xTlJL4YLkLBj Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQTjeVlXFhMdtOerh91nsYzB+azjRQUCXIaVFAAKCRBnsYzB+azj RQgfAP0bi7AfMq7NqN3sXyROU8NfOEq3wUoL0mMbJeRtyPpiTwEA5Hl4jwRQFlp0 7dsvXEhB3TaE5ESXib9dUai7ezTU4wk= =zRfS -----END PGP SIGNATURE----- --Sig_/xfmX_SEqiC4xTlJL4YLkLBj-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:09:35 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88F071526880 for ; Mon, 11 Mar 2019 17:09:35 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 221297085C for ; Mon, 11 Mar 2019 17:09:35 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x82b.google.com with SMTP id s1so5913092qte.5 for ; Mon, 11 Mar 2019 10:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=MuEixTC9KBJk8+8ZW00kG66DYKPwbkqL0B6vVpyCB1g=; b=S51A5ezurLl6xldD0CVgLgpzW5la2fTUbIwDWqLwKZuojMrE0L+kvrZ2BIFF2AFE4l Ub/IgZRo4UExRfZgfFHgL2RpQOwvbb7PpzICFbTD+p7UQbDImT7BOxp+oiSVXBbgbfmy x/BbRN1AVjlqzAY3n+cMRKTUrP1MdTuPo3aoq6+jR1fBdmfsIWjX6v0Q1HN9JqrXYaJO RFhVO3Dng8NcocLhMTV5AZ5HATCaBJ4gSrQyr/jFwrA4AZ5sLKUY1mRyx+zoX1v4R7d2 hcP55YDzwyKbQgt6UG/LU+Qp7sU4fd82BrIXg0cggbfAyKbF522LF+02dilapSguQ9Y5 kdkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=MuEixTC9KBJk8+8ZW00kG66DYKPwbkqL0B6vVpyCB1g=; b=UCVF3UdbOJ82PYFa7sa2p0CNOc9MmIgVXBQk9vNFtKqgTEjgG/V0dnqcTfaEZawTUd VQgfnwIG5FSfD0vxWCoIDkKKspoZyfhfdFnoryjhb+Fi3lQ+Pa2Cq6gP6q8T5qGbvmtv IFBleXi6agjRt4dSzXrO4j/SVs58FPhPGgs2sIf8/LlmgHK8bqB+ty6tvpZ3FdqQVLjl sXwZmdCiTj0kypHFR1THyc+bp3aRKjoPP7wy2jIjlTnuaoQoSR4lXtBDmxcpJ6xttH2C +sVrS5tL0HuMpY9H2cggfVYIQFRTmlxpygwDM1M69k8aL31T7Aidg4Vrj57jMcXKuyg5 uF9g== X-Gm-Message-State: APjAAAX/OSEpjLvzr60AdBuvMy5RPVaqJhPqFUxMBchMH6elagZBze0o wQAAuyHLu2LkO4ZHrQv9Yc0TCQtz0Pwmpw== X-Google-Smtp-Source: APXvYqxugtQ6GeWPdB0rE9ZDmiyfZyeP5lC3ioHJS/oYgwKsVXyFuTeowFH+MD1X8SBZcPMyVPQfSA== X-Received: by 2002:a0c:d2b9:: with SMTP id q54mr26487662qvh.196.1552324174422; Mon, 11 Mar 2019 10:09:34 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.108]) by smtp.gmail.com with ESMTPSA id w20sm3233859qki.35.2019.03.11.10.09.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 11 Mar 2019 10:09:33 -0700 (PDT) Date: Mon, 11 Mar 2019 13:08:53 -0400 From: Shawn Webb To: Mark Raynsford Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311170853.cwjclznvsyd2wksu@mutt-hbsd> References: <20190311170420.61bd34c6@almond.int.arc7.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="c6jwskjot6bhvlcb" Content-Disposition: inline In-Reply-To: <20190311170420.61bd34c6@almond.int.arc7.info> X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 221297085C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; REPLY(-4.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:09:35 -0000 --c6jwskjot6bhvlcb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 11, 2019 at 05:04:20PM +0000, Mark Raynsford via freebsd-virtua= lization wrote: > Hello! >=20 > Leaving aside userland monitoring tools such as Prometheus[0], is there > any way to detect on the host that a guest kernel running in bhyve has > booted? I'm assuming "booted" in this sense to mean "PID 1 has probably > started". I'm guessing that there probably isn't, but I thought I'd > better ask anyway. "Not booted" would mean something like "Is sitting > at the Grub prompt doing nothing". If your guest OS supports it, you could probably write two scripts that uses virtio_console(4), one for the guest to tell the host "HELLO" and one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is received. Of course, replace the "HELLO" and "NICE TO SEE YOU!" with the right logic you're looking for. ;) The "HELLO" bit could be written as an init script. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --c6jwskjot6bhvlcb Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlyGliEACgkQaoRlj1JF bu7Nkw/9EjB8VmUIM19JMRZeeysuzhy1VHu4pB4NWmFYFOTUYOpgJomb/m1q3fuz xPP/GWoe9pkn3ktXImt1PnyVHJDWUn9c0pY+v4Yt8EVqlW5R6qTxcU9fIoDDcnwv xWg/WVAzXUPO+pfCDslyacXW2P4s9fQY83bBy4W7Z33sfMSNv2tUTdc7v27c7kMm iJ7rRi8Hx2ifMa7ktWZXs96XMi6NLtxeoyDbTy1Iu7pWxlL9d6tD76IWcdaFKS1N a0yOJDBt1F3UTQg25Pg2FlzBHF+wAsFfb/JCWSHQgRkWfJwgMrfuJTdJeH26HNDE p4J/JZYTSc4tCldR5vKkqQzSHcgfLR3MxAob6jjWpqIHDzFoHZx1lfld4IGezeZd Mt+4nic/M7QPB3vhL/rRMSXK0KgAgADp/vqj4X4FLiElvTIwxvWR5OuPnSyrhbwr JgGmbVMtyXJ9jxtDaXz5uiZ3d/pr2CFXBYQcg2QxbS/pC/rWHFK0cYVrWfwalqHx FT9ojNU2uVKO3YlDCotV3D8S5SzawcNi/oJbSyp32dNltMLM9Ivun+sATQfJPmpi cYUHJb3NSqh7YuIyHneP7pzJhbkvlkH8kk3xR/Xtr7Bb1H1F8FnilF+qjSYNDbtc nYxTZflu11DmEyCoTmNx/mF34SNF7sLK/mzJslvee/e/Q09lVfI= =doqX -----END PGP SIGNATURE----- --c6jwskjot6bhvlcb-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:09:48 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2625B15268B4 for ; Mon, 11 Mar 2019 17:09:48 +0000 (UTC) (envelope-from jason@corrupted.io) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C135C7088D for ; Mon, 11 Mar 2019 17:09:47 +0000 (UTC) (envelope-from jason@corrupted.io) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id CAC6D254E8 for ; Mon, 11 Mar 2019 13:09:46 -0400 (EDT) Received: from imap3 ([10.202.2.53]) by compute6.internal (MEProxy); Mon, 11 Mar 2019 13:09:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corrupted.io; h= message-id:in-reply-to:references:date:from:to:subject :content-type; s=mesmtp; bh=A3W3eU8fcMn8ud6MI3A3vLiB1CAR/AGB4ETE CZwF7+E=; b=Kz7mTumKiMMdgDPvCDq8+xylk9MTTfRb5G89CVS1ZcGMzBOogD6U yclw2J459XiuXFA6Fx4rzrt5TGSBeJ26qxdd25vdAqLZhZRpUNAtTobXgqIWwLuf JPJWrUJ1pdduF8nsG3gnEng9m9KFQyZyLxLL+OZYk5oZPXXqYdhrrWo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:references:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=A3W3eU8fcMn8ud6MI 3A3vLiB1CAR/AGB4ETECZwF7+E=; b=UKWguJNAKuv4Jy0hXHPQfxmEZlCk+uX2E kegOZS6qFDsxHHARVgFWnrEzRHXCjQZ+z+bfxmCpW7jXXw2LblrQnzYSm8B0AFEq +SmZ91GERLZFqOSsEa4x5mQQwHNL92peT5+fcvobswxVpH1ispGeTT1arpWtiMF+ aqGiJE4PBXp+dPjJntlyLGpQ0IyzDvCKEjVH5sjTRTCFRBUDx43cYZUM94VGJk9+ XssWuX+Yjlgwd7YCy2DFngXpa/apMHeUIdixG/8XzI7xt7bZNoM/zQRLVdvHDZG1 mEIAOj/UkUl/Kx1eEEbOuReerHm6c3Z8fPZ7DMsD02DbA4WfbEbsg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrgeeigdellecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfkfgjfhffhffvufgtsehttdertd erredtnecuhfhrohhmpedflfgrshhonhcuuegrrhgsihgvrhdfuceojhgrshhonhestgho rhhruhhpthgvugdrihhoqeenucffohhmrghinhepphhrohhmvghthhgvuhhsrdhiohdpih hojehmrdgtohhmnecurfgrrhgrmhepmhgrihhlfhhrohhmpehjrghsohhnsegtohhrrhhu phhtvggurdhiohenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 4A0FC7C16B; Mon, 11 Mar 2019 13:09:46 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.5-925-g644bf8c-fmstable-20190228v5 X-Me-Personality: 64405576 Message-Id: <09d8981f-6dae-4c84-a1e4-0c9f4aa5cc41@www.fastmail.com> In-Reply-To: <20190311170420.61bd34c6@almond.int.arc7.info> References: <20190311170420.61bd34c6@almond.int.arc7.info> Date: Mon, 11 Mar 2019 13:09:45 -0400 From: "Jason Barbier" To: freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Content-Type: text/plain X-Rspamd-Queue-Id: C135C7088D X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.989,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:09:48 -0000 None I know off the top of my head nor would that be really something easily done without some sort of agent. We could probably make some sort of inference based calls to the emulated registers but even then that wouldn't be that correct. --- Jason Barbier | E: jason@corrupted.io On Mon, Mar 11, 2019, at 10:05 AM, Mark Raynsford via freebsd-virtualization wrote: > Hello! > > Leaving aside userland monitoring tools such as Prometheus[0], is there > any way to detect on the host that a guest kernel running in bhyve has > booted? I'm assuming "booted" in this sense to mean "PID 1 has probably > started". I'm guessing that there probably isn't, but I thought I'd > better ask anyway. "Not booted" would mean something like "Is sitting > at the Grub prompt doing nothing". > > [0] https://prometheus.io/ > > -- > Mark Raynsford | http://www.io7m.com > > > Attachments: > * null From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:22:12 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B9AE1527086 for ; Mon, 11 Mar 2019 17:22:12 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from mail.io7m.com (mail.io7m.com [IPv6:2001:19f0:5:752:f000::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.io7m.com", Issuer "arc7 CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E63EC710AC for ; Mon, 11 Mar 2019 17:22:11 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from almond.int.arc7.info (cust187-dsl61.idnet.net [212.69.61.187]) by mail.io7m.com (Postfix) with ESMTPSA id BA96A2F2B6; Mon, 11 Mar 2019 17:22:09 +0000 (UTC) Date: Mon, 11 Mar 2019 17:21:58 +0000 From: Mark Raynsford To: Shawn Webb Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311172158.1c06b5d9@almond.int.arc7.info> In-Reply-To: <20190311170853.cwjclznvsyd2wksu@mutt-hbsd> References: <20190311170420.61bd34c6@almond.int.arc7.info> <20190311170853.cwjclznvsyd2wksu@mutt-hbsd> Organization: io7m.com OpenPGP: id=E379595716131DB4E7AB87DD67B18CC1F9ACE345; url=http://io7m.com/pgp/E379_5957_1613_1DB4_E7AB_87DD_67B1_8CC1_F9AC_E345.key MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/5ZHrWT4UeBmD_GbwcW82hXA"; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:22:12 -0000 --Sig_/5ZHrWT4UeBmD_GbwcW82hXA Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On 2019-03-11T13:08:53 -0400 Shawn Webb wrote: > > If your guest OS supports it, you could probably write two scripts that > uses virtio_console(4), one for the guest to tell the host "HELLO" and > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is > received. >=20 They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing one out of three of those supports it... I suppose my other option would be to add (another) NFS mount in each guest, and have them touch a file early in the init script (and possibly touch a different file early in the shutdown script). --=20 Mark Raynsford | http://www.io7m.com --Sig_/5ZHrWT4UeBmD_GbwcW82hXA Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQTjeVlXFhMdtOerh91nsYzB+azjRQUCXIaZNgAKCRBnsYzB+azj RS/0AQDn11JBFpowED1iZMoxWXFIQT41TwTUzKgun1vWUk+21QEA1IJg6O8MV0XL dSf0pzhZBftI5n9uAcJBUm8pf8eNzQY= =HbHx -----END PGP SIGNATURE----- --Sig_/5ZHrWT4UeBmD_GbwcW82hXA-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:28:05 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9964152715A for ; Mon, 11 Mar 2019 17:28:04 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70200713DA for ; Mon, 11 Mar 2019 17:28:04 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x842.google.com with SMTP id d2so5961946qti.11 for ; Mon, 11 Mar 2019 10:28:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=owhunB5TFLEKrJqNB8M5OLnrrb8EEztKSWmlzAtsaT8=; b=BR18pXjbyRoHfJvz+gsRL/59JyAEDXs66q2a4BsGINXt3HrOm6v9Bi7rbetWAdXyai +9sX3xOWc7jYEvg80dYeJzDekXTpZhLDRPPd/kwFxkQymVaCMAX6RtfKqv/MBN5pQ5f4 KOI9ztC90hY9361w8oCZ1WDSw+WXI4onssWvaOLSzuYA4tCO8fiOrrfryzaKQnU9SuCy mE0oDbTCvaAhUyJK2m7eC7u/VB47uI4l16QL68aEf0RyysgGer6ltK8vuPaeYsEzgTyj 26zzRaMo0pxT3pYjkDMZhi+0llPD4IFWnrXQGbKEmdNk3S11BLhyjdkSvSp5JxKTTLrV GfTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=owhunB5TFLEKrJqNB8M5OLnrrb8EEztKSWmlzAtsaT8=; b=a8J7/N8RyAFBn0FkwCP623c9nmsYe+WA5bhXdOCyDOiWMZ3yPAnW6t+/QJLQnW6NSJ kQj/Jsip/9ZICzJ1mOlpuqfAUqUMNR6CBIHfeZLfk4MwG7vyILuBKxjnVqOCoQsZAzkk FkORL3JRhP/gOPTMUUKkCObsKcQwJb5HMel+wrHMi6YwzfxqHCUQHphFneDg2zKyGh02 B2p58hUh7HVpLcWneQIs7/mmltbByjDaYr/937L3oCCajG2381mvjLy+kmamfh7vx0LU o6fDB3k7n1S2oDk+rYrg/9RUqXuqX6ch5KbS6YrmfhQgZFqfACN3SvPcP3oPRHK+yiQ+ rGjQ== X-Gm-Message-State: APjAAAVr16Dt2naWKtOSNriu8sd/cgVGP1hC69nMRl5uXRIpUHnBOrji oGTcMzetAV9raGm8KOszzvHF+kt4tcUiqA== X-Google-Smtp-Source: APXvYqxVyD75LncV4F3Sl5X+XW4PuRkpf9YN78pwN8dkELB0AMnQjmq9h/VYsKf5UzVfkGdLp2B18g== X-Received: by 2002:ac8:1209:: with SMTP id x9mr25524009qti.199.1552325283677; Mon, 11 Mar 2019 10:28:03 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.108]) by smtp.gmail.com with ESMTPSA id p189sm3685556qkf.20.2019.03.11.10.28.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 11 Mar 2019 10:28:02 -0700 (PDT) Date: Mon, 11 Mar 2019 13:27:23 -0400 From: Shawn Webb To: Mark Raynsford Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311172723.idvwvphb4acarxvi@mutt-hbsd> References: <20190311170420.61bd34c6@almond.int.arc7.info> <20190311170853.cwjclznvsyd2wksu@mutt-hbsd> <20190311172158.1c06b5d9@almond.int.arc7.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rhjfcmgrep4mtrv5" Content-Disposition: inline In-Reply-To: <20190311172158.1c06b5d9@almond.int.arc7.info> X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 70200713DA X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:28:05 -0000 --rhjfcmgrep4mtrv5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 11, 2019 at 05:21:58PM +0000, Mark Raynsford wrote: > On 2019-03-11T13:08:53 -0400 > Shawn Webb wrote: > > > > If your guest OS supports it, you could probably write two scripts that > > uses virtio_console(4), one for the guest to tell the host "HELLO" and > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is > > received. > >=20 >=20 > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing > one out of three of those supports it... >=20 > I suppose my other option would be to add (another) NFS mount in each > guest, and have them touch a file early in the init script (and > possibly touch a different file early in the shutdown script). Both FreeBSD and Linux supports virtio_console(4). I have no idea about OpenBSD, but I'm sure they'd be open to an implementation if asked. The NFS solution would work, but it would be somewhat fragile. What happens when a VM crashes? What happens when the host crashes? Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --rhjfcmgrep4mtrv5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlyGmnQACgkQaoRlj1JF bu6f/Q/6A4f807voEYU4VNzsq+ubJxGREEVRgswN+AjB4T5afwHFVkVmpbcjIjRc JbVTNhV2oxOLjPBoG1A9rm6uf/YIcybStAAHNluEJ08XQDwcpjTuYqgtCXKkGjqj vuAuubHdMiS2LN7pcv23rinhlNylkn2a7Oogf5SFaeZ4B12R/YfhYoUCt58Tjrzn ZVBlkwlKVLdSyyxCSpyYg+Bqb9WSB9Dgl6W1uMk9YzfbhIhFHwNUvQ+lT3xFViFh iSFjjW2Dk+TpvwQH3ErNNEyeK5cuA7gYFUltMcAzRh6EZVnLiaOnD4REtUcT8hQs HwViU3LiF4AxNRabZ/z895j3+vkQ2rMdK+w2eq7hNlvo0f7i/BMXbgmsw9iRRWcM uWRCxdb54e1/mwsle0kBN0yu8qn7Sl2HEHi0CFCQCE5tqSI9QcxSb2qwJoD4kv5v v/wPtPOn1QpoJ4uwdFwUF2WUY+zRe06M/DeYwA2sMN6GXewQtnqvB2hHr1vYiLoO aJ6fJ5emZyGteL71aFUmKq4dE8nbhk2vzvbFKAHp+VhxEIJ+NVuK9lY2LoNpURtt GS6nXcEmnocMWoJAK714im1pxV/RoTyQ0sq7N+Oke3iErwQtLcxjAQnGUzjNI5yi JfrflzfTKM4uY8e5mPh8UDISMVWivh4CUzgbpuVfFmw3Lc8HDsA= =qVyS -----END PGP SIGNATURE----- --rhjfcmgrep4mtrv5-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:46:48 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D27521527CAB for ; Mon, 11 Mar 2019 17:46:47 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from mail.io7m.com (mail.io7m.com [IPv6:2001:19f0:5:752:f000::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.io7m.com", Issuer "arc7 CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D08F72274 for ; Mon, 11 Mar 2019 17:46:47 +0000 (UTC) (envelope-from list+org.freebsd.virtualization@io7m.com) Received: from almond.int.arc7.info (cust187-dsl61.idnet.net [212.69.61.187]) by mail.io7m.com (Postfix) with ESMTPSA id 2A7AA2F2D7; Mon, 11 Mar 2019 17:46:45 +0000 (UTC) Date: Mon, 11 Mar 2019 17:46:35 +0000 From: Mark Raynsford To: Shawn Webb Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311174635.63a071e2@almond.int.arc7.info> In-Reply-To: <20190311172723.idvwvphb4acarxvi@mutt-hbsd> References: <20190311170420.61bd34c6@almond.int.arc7.info> <20190311170853.cwjclznvsyd2wksu@mutt-hbsd> <20190311172158.1c06b5d9@almond.int.arc7.info> <20190311172723.idvwvphb4acarxvi@mutt-hbsd> Organization: io7m.com OpenPGP: id=E379595716131DB4E7AB87DD67B18CC1F9ACE345; url=http://io7m.com/pgp/E379_5957_1613_1DB4_E7AB_87DD_67B1_8CC1_F9AC_E345.key MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/T2w0M3ABGWkJVOl+1e70YTY"; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:46:48 -0000 --Sig_/T2w0M3ABGWkJVOl+1e70YTY Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On 2019-03-11T13:27:23 -0400 Shawn Webb wrote: > > Both FreeBSD and Linux supports virtio_console(4). I have no idea > about OpenBSD, but I'm sure they'd be open to an implementation if > asked. Right. >=20 > The NFS solution would work, but it would be somewhat fragile. What > happens when a VM crashes? What happens when the host crashes? At least in my case: If the VM crashes, it'll be restarted by a process supervisor (runit, here). If the host crashes, I likely have bigger problems. In any case, I think that's still fine because all the host would care about is if the guest's file was touched more recently than the last time the host tried to start a bhyve process for it. I suppose I should elaborate a bit: I do have monitoring via Prometheus in place, but I'd like to try to stagger VM startups a little as starting up a lot of them in parallel on boot tends to overwhelm the machine slightly. Once they're all up and running in a steady state, things are fine. I would typically stagger the startup of ordinary services (inside a jail, for example) by using dependencies in runit - it has a facility to pause a service until a dependent service has been started. Unfortunately, that can't work in this case because once the bhyve process has been started, the host can't tell if/when the guest has actually fully started up. As far as runit is concerned, the service is up and so any dependent services should be started too. -- Mark Raynsford | http://www.io7m.com --Sig_/T2w0M3ABGWkJVOl+1e70YTY Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQTjeVlXFhMdtOerh91nsYzB+azjRQUCXIae+wAKCRBnsYzB+azj RXasAP0Y9Gx4iHbA6o1gf46xtluFg1vbL1/HLyljnCCFVC4O/wEAoPd0ANkSb+Mz PGN/ayvBSmlq1R3kSQ0joTT+HMiIvQA= =msQ8 -----END PGP SIGNATURE----- --Sig_/T2w0M3ABGWkJVOl+1e70YTY-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 17:58:59 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6230E1527FDD for ; Mon, 11 Mar 2019 17:58:59 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2E4372872 for ; Mon, 11 Mar 2019 17:58:58 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x2BHwtKh010699; Mon, 11 Mar 2019 10:58:55 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x2BHwttd010698; Mon, 11 Mar 2019 10:58:55 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201903111758.x2BHwttd010698@ gndrsh.dnsmgr.net> Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: <20190311172158.1c06b5d9@almond.int.arc7.info> To: Mark Raynsford Date: Mon, 11 Mar 2019 10:58:55 -0700 (PDT) CC: Shawn Webb , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: C2E4372872 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.990,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 17:58:59 -0000 -- Start of PGP signed section. > On 2019-03-11T13:08:53 -0400 > Shawn Webb wrote: > > > > If your guest OS supports it, you could probably write two scripts that > > uses virtio_console(4), one for the guest to tell the host "HELLO" and > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is > > received. > > > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing > one out of three of those supports it... > > I suppose my other option would be to add (another) NFS mount in each > guest, and have them touch a file early in the init script (and > possibly touch a different file early in the shutdown script). Well ICMP is in the kernel, and should be working as soon as the interface is up, long before you could do anything with NFS, so rather than the complexity above a simple ping would suffice. There is also the phase of vmm(8) startup that when you are running bhyveload vs bhyve and iirc grubload vs bhyve, that can be detected. vmbhyve does so and says you are in state looader when you do a vm list. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Mar 11 18:03:02 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1F5515282F8 for ; Mon, 11 Mar 2019 18:03:02 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C41572CDB for ; Mon, 11 Mar 2019 18:03:01 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x2BI30u0010733; Mon, 11 Mar 2019 11:03:00 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x2BI2xTK010732; Mon, 11 Mar 2019 11:02:59 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201903111802.x2BI2xTK010732@ gndrsh.dnsmgr.net> Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: <20190311174635.63a071e2@almond.int.arc7.info> To: Mark Raynsford Date: Mon, 11 Mar 2019 11:02:59 -0700 (PDT) CC: Shawn Webb , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 0C41572CDB X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.995,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 18:03:02 -0000 > On 2019-03-11T13:27:23 -0400 > Shawn Webb wrote: > > > > Both FreeBSD and Linux supports virtio_console(4). I have no idea > > about OpenBSD, but I'm sure they'd be open to an implementation if > > asked. > > Right. > > > > > The NFS solution would work, but it would be somewhat fragile. What > > happens when a VM crashes? What happens when the host crashes? > > At least in my case: > > If the VM crashes, it'll be restarted by a process supervisor (runit, > here). I think ping would suffice to make that determination? > If the host crashes, I likely have bigger problems. In any case, I > think that's still fine because all the host would care about is if the > guest's file was touched more recently than the last time the host > tried to start a bhyve process for it. You could inside the vm simply touch /tmp/foo from cron and from the host see that the diskimage last modified time updated. Assuming some cacheing does not get in the way. > I suppose I should elaborate a bit: I do have monitoring via Prometheus > in place, but I'd like to try to stagger VM startups a little as > starting up a lot of them in parallel on boot tends to overwhelm the > machine slightly. Once they're all up and running in a steady state, > things are fine. I would typically stagger the startup of > ordinary services (inside a jail, for example) by using dependencies in > runit - it has a facility to pause a service until a dependent service > has been started. Unfortunately, that can't work in this case because > once the bhyve process has been started, the host can't tell if/when > the guest has actually fully started up. As far as runit is concerned, > the service is up and so any dependent services should be started too. The package vmbhyve has starggered startup in a specific ordered list implemented. It is all writtin in /bin/sh, so easy to adapt. > Mark Raynsford | http://www.io7m.com -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Mar 11 18:04:42 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B566152837C for ; Mon, 11 Mar 2019 18:04:42 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3FDFF72D83 for ; Mon, 11 Mar 2019 18:04:41 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x842.google.com with SMTP id b3so6131216qtj.3 for ; Mon, 11 Mar 2019 11:04:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=VAVh5MWXKf042KyCbJRfBSiwWOXmDwb/LHNPTXV01wA=; b=jR/XE3iv1lP12hwpbo9BzX9+TJxXChP0c9/no1Hr96Dluyis16E/VDXOVSMptWqxqZ O/umKDzMzuvYgPePHoKK93x/R0PEdY5iL2g7+4k/F8i3zREt9SdFWoHv1IXgceB5vpdT YosmhuSpklMjtP8HS9LDzFb8dDTGnlZg5VX8CMKecxaD9lKY+A1+KgjIlFixjRKkR6NN yqHMxzgCG2HbneAX5yIh6OfFkpznEvdm0pKpgvplUZdPTLx+g3L0ngmV6fCGmiTr0ZHP 0iRGzmn4oSGMaIHqqvnRXD9xl2BQ57HWd9PoNO19e1RSe7eVem3VdqUC2lqeNqU71e+P rH6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=VAVh5MWXKf042KyCbJRfBSiwWOXmDwb/LHNPTXV01wA=; b=sSN0/YQzET2BrkB9TpzxGN8kW2NJbjvcALIYdOQNKKUQROI2ZU4yz7y8VnnfEtdaqn Q9wl3Ff7gmtYb/ilG5uB//Cdir2IAXEZ4Lrm0zfYA+PzHwNVy9vY5tZYW6fK/kykrlyK 5hQ2dmnaa0nA+vCEWM8P1GWXVtQsZztFXH0ZqhKkJkxt7W5EaaDKA89lkjRn+VwFDyG4 78eFLVe6DwF/kvqeVg7nNBpEv/R2neOlxETxMhc2VeYWBne9NN6dgEXi2WiHG9OKY5Pm HgPY1LUw+rV2UWtn2vezjV/tzkxkbE/DLpm/iC4MVuURuOP6BrZeL/aBbUw4kgVhFDtn hE+g== X-Gm-Message-State: APjAAAUAf7/iWr1yWQ50hUjYELq1AeVJ8/DhQhBCvC3OMvvv9W/CX1Du mQ4LoyBGerMKSGtjXih8GyOtsQ== X-Google-Smtp-Source: APXvYqwIvn3zoWNnT6r/wNjDH8CIxWuj2Z8ucMUtt5TDkiPw+bvCQLTssnuZhUKpFDgE609SZHLAtg== X-Received: by 2002:aed:3084:: with SMTP id 4mr26507614qtf.19.1552327480610; Mon, 11 Mar 2019 11:04:40 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.108]) by smtp.gmail.com with ESMTPSA id 5sm4185622qtw.69.2019.03.11.11.04.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 11 Mar 2019 11:04:39 -0700 (PDT) Date: Mon, 11 Mar 2019 14:03:59 -0400 From: Shawn Webb To: "Rodney W. Grimes" Cc: Mark Raynsford , freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311180359.knal2sjbmxlgzduh@mutt-hbsd> References: <20190311172158.1c06b5d9@almond.int.arc7.info> <5c86a1e4.1c69fb81.4c2f8.a040SMTPIN_ADDED_BROKEN@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="dszkua3ez6tmcq3x" Content-Disposition: inline In-Reply-To: <5c86a1e4.1c69fb81.4c2f8.a040SMTPIN_ADDED_BROKEN@mx.google.com> X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 3FDFF72D83 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=jR/XE3iv; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::842 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-5.57 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx3.googlemail.com]; NEURAL_HAM_SHORT(-0.82)[-0.823,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.64)[ip: (1.68), ipnet: 2607:f8b0::/32(-2.74), asn: 15169(-2.07), country: US(-0.07)]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 18:04:42 -0000 --dszkua3ez6tmcq3x Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote: > -- Start of PGP signed section. > > On 2019-03-11T13:08:53 -0400 > > Shawn Webb wrote: > > > > > > If your guest OS supports it, you could probably write two scripts th= at > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is > > > received. > > >=20 > >=20 > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing > > one out of three of those supports it... > >=20 > > I suppose my other option would be to add (another) NFS mount in each > > guest, and have them touch a file early in the init script (and > > possibly touch a different file early in the shutdown script). >=20 > Well ICMP is in the kernel, and should be working as soon as the > interface is up, long before you could do anything with NFS, > so rather than the complexity above a simple ping would suffice. Just a note: Windows systems disable inbound ICMP by default, but inbound ICMP support can be enabled post-installation. > There is also the phase of vmm(8) startup that when you are > running bhyveload vs bhyve and iirc grubload vs bhyve, that > can be detected. vmbhyve does so and says you are in state > looader when you do a vm list. I would suggest using bhyve with UEFI. I wish a death upon bhyveload and grub2-bhyve. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --dszkua3ez6tmcq3x Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlyGowkACgkQaoRlj1JF bu6j3g/9FBifnhcnN1ti121uKjuswOO52dX2q1SVcX0wjynAPZJL23x9KA1pKHbX MMNNmPmGRoeh0/vQSM4MGwV6WJrDo4wPYWdMCjis/G/d1Wb6VLJO9I3enk0/w73X V18QK5IPy+i7hw8Q2UgfvkTmF2viE2rc8B/AvPJ63sN7N/JXgAa/TPE6Kt585wOD S6He54wib2vslJmchY2gKVyrXUB0QwkTEngy9a2+GAcff1tmWQO+vt2kg6mjpzbl LN0g8ibW2v0hUxIr4G2dODErF60ABaLQNw3zDma/Cco043E2w+aNAhWtPGdbqTKM 0htEKx7arBS7PQo6jd11wBc9fA35/9uq2nu7A0NelRVbOLZoYaXlhpsdmQqQucYb kEqzUNzLPPIjpzXgNMcTP9FOy1ZIF1s8CvNfYAzFHQl6EwXa4nWyl3+ZRPnrv/bH QHPbMIbnqqkkWQbLzoV7FLtLqGflLR5+QW/oTGuaMhg1A4604hn3rmPGa/iaM2dc JUn7/LDpz9Ksaj7O2ELWeT6YfU/9QszTtM8bsO7BWoY5j2UeSwOoqF6dgnCZLPNf zXX5XM7NAIkevaJkkiWSsgSyayre/3cTs9UAJaRt8WN76eXYP8vKuwjIyZ3kLoEm KyH/RuyyMVAZHjFbWbmqZ0JJMwOKCDsnoacI208ed499j3z4awk= =s8Az -----END PGP SIGNATURE----- --dszkua3ez6tmcq3x-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 18:09:17 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CF4E1528704 for ; Mon, 11 Mar 2019 18:09:17 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08C937398F for ; Mon, 11 Mar 2019 18:09:15 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x2BI97hN010769; Mon, 11 Mar 2019 11:09:07 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x2BI974C010768; Mon, 11 Mar 2019 11:09:07 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201903111809.x2BI974C010768@ gndrsh.dnsmgr.net> Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: <20190311180359.knal2sjbmxlgzduh@mutt-hbsd> To: Shawn Webb Date: Mon, 11 Mar 2019 11:09:07 -0700 (PDT) CC: "Rodney W. Grimes" , Mark Raynsford , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 08C937398F X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [3.90 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.21)[0.210,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.94)[0.935,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: gndrsh.dnsmgr.net]; NEURAL_SPAM_LONG(0.16)[0.158,0]; INVALID_MSGID(1.70)[]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(0.00)[ip: (0.05), ipnet: 69.59.192.0/19(0.03), asn: 13868(0.01), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 18:09:17 -0000 > On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote: > > -- Start of PGP signed section. > > > On 2019-03-11T13:08:53 -0400 > > > Shawn Webb wrote: > > > > > > > > If your guest OS supports it, you could probably write two scripts that > > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and > > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is > > > > received. > > > > > > > > > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing > > > one out of three of those supports it... > > > > > > I suppose my other option would be to add (another) NFS mount in each > > > guest, and have them touch a file early in the init script (and > > > possibly touch a different file early in the shutdown script). > > > > Well ICMP is in the kernel, and should be working as soon as the > > interface is up, long before you could do anything with NFS, > > so rather than the complexity above a simple ping would suffice. > > Just a note: Windows systems disable inbound ICMP by default, but > inbound ICMP support can be enabled post-installation. > > > There is also the phase of vmm(8) startup that when you are > > running bhyveload vs bhyve and iirc grubload vs bhyve, that > > can be detected. vmbhyve does so and says you are in state > > looader when you do a vm list. > > I would suggest using bhyve with UEFI. I wish a death upon bhyveload > and grub2-bhyve. I have no love for them either, but until we get our UEFI updated it is a rather sad state of affairs. If we could a) get on a modern version of ed2k, and b) bet the CSM fixed so that we could actually boot bios mode stuff with it and c) maybe make a port of seabios that could be used then we would be kicking some seriuos stuff! > Thanks, > Shawn Webb -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Mar 11 18:13:35 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE2891528962 for ; Mon, 11 Mar 2019 18:13:35 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 97F8773D18 for ; Mon, 11 Mar 2019 18:13:34 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk1-x744.google.com with SMTP id z76so2444655qkb.12 for ; Mon, 11 Mar 2019 11:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=l5RAA+jSpXlIj7dLOMKqtu3qL4KGQU2seJCecRYjba4=; b=JhFHQG0hU0KJSi0XkdA387mro0Ep20RNd++mkJy/w1hwBwRZDpe2FccNiYvwVNgKS+ liAu/JX6OAM8SF+Gfbv2HNiiu80IHmLL51pdeXylbaXYkciF5C7RMYO/EitLREoIOMPm xqn4NIxMxSLDqFVuEKvmZ5AJYiBFtsPW9KRaSqJ4ruaXEpO9Y32vv6G+hyVQCLfPJBoY 6VXsZsiMkYEKmVwjmz9gR3jfjhoq5uVnOPr/QlvLx9Ux6B5Bh6Hh8sw37ZVHpkIDxde5 P95Y6I9HnFJi1ZirK99FSZx4Y9lvNaCUxFN46i+pC03fSNj3V4+/5mGs53xKe1QTBCf7 yBgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=l5RAA+jSpXlIj7dLOMKqtu3qL4KGQU2seJCecRYjba4=; b=d5zsXv4YNnDnrz1P5uS39kYciYbbtj8snlUQTE5qUVp/t8ozFNvTi3SnMuTBzz+K6t Hs8jAYHyKuWcY3SzHj/ie67xqpneIqK0X9IE+n/wisULBrk5+oYBP9N70Zcirpupmqxm f2gtZgr9x5qNSAAhLUyCimG6IwcWCDG4rxHHM8EcR0htiopCrcYSfMvFCjVGSoO6WU4c hBTwDmFbHZ4VMfhVrhf414yRTQ+LKBtrl0/wAVSylUwRrrvSj+eoHwaAX7aMgApfqUSG pyLH7C+jk0xyVZoFXKoAmsFbF0y/OhApRbT708Ej37VyU0QeLYrqQrKmRRCPVSvGNieY Fqdg== X-Gm-Message-State: APjAAAU3bnVa03L1/1Yt9NtOidHZJ8YfOWVXgiwIzeo0p11sqZIkNd/d etjmM57d23UmBPVHPOkwUT6brw== X-Google-Smtp-Source: APXvYqwjzsE0PiCHn46yS0cupgOZHxhugLWCsKgPc91eSQzyDvz62s/JRdywyXYeavmp9OxhLrLqzA== X-Received: by 2002:a37:c384:: with SMTP id r4mr6332096qkl.306.1552328013488; Mon, 11 Mar 2019 11:13:33 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.108]) by smtp.gmail.com with ESMTPSA id c19sm3605581qkg.88.2019.03.11.11.13.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 11 Mar 2019 11:13:32 -0700 (PDT) Date: Mon, 11 Mar 2019 14:12:52 -0400 From: Shawn Webb To: "Rodney W. Grimes" Cc: Mark Raynsford , freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted Message-ID: <20190311181252.4xrwzfidzfhsad46@mutt-hbsd> References: <20190311180359.knal2sjbmxlgzduh@mutt-hbsd> <5c86a44e.1c69fb81.cd645.9e28SMTPIN_ADDED_BROKEN@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ulmgg6hb4u5r6frw" Content-Disposition: inline In-Reply-To: <5c86a44e.1c69fb81.cd645.9e28SMTPIN_ADDED_BROKEN@mx.google.com> X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 97F8773D18 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=JhFHQG0h; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::744 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-5.48 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; NEURAL_HAM_SHORT(-0.80)[-0.799,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.58)[ip: (1.99), ipnet: 2607:f8b0::/32(-2.74), asn: 15169(-2.07), country: US(-0.07)]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 18:13:36 -0000 --ulmgg6hb4u5r6frw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 11, 2019 at 11:09:07AM -0700, Rodney W. Grimes wrote: > > On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote: > > > -- Start of PGP signed section. > > > > On 2019-03-11T13:08:53 -0400 > > > > Shawn Webb wrote: > > > > > > > > > > If your guest OS supports it, you could probably write two script= s that > > > > > uses virtio_console(4), one for the guest to tell the host "HELLO= " and > > > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELL= O" is > > > > > received. > > > > >=20 > > > >=20 > > > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessi= ng > > > > one out of three of those supports it... > > > >=20 > > > > I suppose my other option would be to add (another) NFS mount in ea= ch > > > > guest, and have them touch a file early in the init script (and > > > > possibly touch a different file early in the shutdown script). > > >=20 > > > Well ICMP is in the kernel, and should be working as soon as the > > > interface is up, long before you could do anything with NFS, > > > so rather than the complexity above a simple ping would suffice. > >=20 > > Just a note: Windows systems disable inbound ICMP by default, but > > inbound ICMP support can be enabled post-installation. > >=20 > > > There is also the phase of vmm(8) startup that when you are > > > running bhyveload vs bhyve and iirc grubload vs bhyve, that > > > can be detected. vmbhyve does so and says you are in state > > > looader when you do a vm list. > >=20 > > I would suggest using bhyve with UEFI. I wish a death upon bhyveload > > and grub2-bhyve. >=20 > I have no love for them either, but until we get our UEFI updated > it is a rather sad state of affairs. If we could a) get on a modern > version of ed2k, and b) bet the CSM fixed so that we could actually > boot bios mode stuff with it and c) maybe make a port of seabios > that could be used then we would be kicking some seriuos stuff! If I'm able to run an internship this year like I did last year, I might have my interns update our UEFI firmware to the latest release and/or make a BSDL seabios port. My employer was recently acquired, so I'm likely not going to run an internship this year. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --ulmgg6hb4u5r6frw Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlyGpSQACgkQaoRlj1JF bu6A/Q/+M0VcP/eKL9RYHZZwkMDqOookRMXoiDnyBulxDZoAPa61KPdeMOcMZ+Yk MtL+l48+xj0JDZparXQZCYsQXPwrIJWbKvu/MZqk303ZeuKMD82SJDLs3709NKx7 bg461K155d6ElT2JZaRZxQ03JoIe+P6d5+hp23QHnjw18j9NK7zJhfX6b2HnxTEP P96n88jhGwAf3cuWykN7JqksrKF8cqm9ntJSPlAiEyma5wFQdeu9IrR05KXVjoc2 yXSJPd5lGdR/Qmdzkioaw4kObN9SFO4QlNKw5A6FsQnU2iBqu39SBx9ULw4fcqqQ ba6loxnU0b+ySVRX1kJyxhLZRiohA3TK5LAncj6M5TNiKObuxU0nGyKaWrZWxwHG aQxTkU3cD/FZnNIcvNta6uskgAVVJTBBcnkQLwJFd6DkvxYU549Esmtvnvx9tk2s UkossC3jzhda9LjMXGkmPwnC9RuzzVOrsw66Q1N+L2Y+Z7xPXz280FHZxjogBcEq 8BCaaQNoHbF4raqoEUw5v619lcNlx0KnoVY6GRf+d5WsVA6zDavpyHLJ7oc5BKmH /e5zxyvJdsNwjTljdPdLtSFC9b1agoFtsahK/HGO7tShoYIsuFHw4uO4fYSpyv1y VlZz3YONIhDgZPooFBBgaNq+Ix34CLL3DfojQvEZYy9+c38yXcA= =TpxC -----END PGP SIGNATURE----- --ulmgg6hb4u5r6frw-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 19:09:45 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18A20152A597 for ; Mon, 11 Mar 2019 19:09:45 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C16B76660 for ; Mon, 11 Mar 2019 19:09:44 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2BJ9e7V076321; Mon, 11 Mar 2019 20:09:40 +0100 (CET) Received: from [217.29.46.2] (unassigned [217.29.46.2] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2BJ9dQf081584; Mon, 11 Mar 2019 20:09:40 +0100 (CET) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Message-Id: <67EAE893-5451-4D37-A129-37AB858C4F26@punkt.de> Content-Type: multipart/signed; boundary="Apple-Mail=_976E1CCC-E48D-448E-8388-45E304997A64"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: bhyve: Detecting that a guest kernel has booted Date: Mon, 11 Mar 2019 20:09:33 +0100 In-Reply-To: <20190311170420.61bd34c6@almond.int.arc7.info> Cc: freebsd-virtualization@freebsd.org To: Mark Raynsford References: <20190311170420.61bd34c6@almond.int.arc7.info> X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 7C16B76660 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[org.freebsd.virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 19:09:45 -0000 --Apple-Mail=_976E1CCC-E48D-448E-8388-45E304997A64 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi all, > Am 11.03.2019 um 18:04 schrieb Mark Raynsford via = freebsd-virtualization : > Leaving aside userland monitoring tools such as Prometheus[0], is = there > any way to detect on the host that a guest kernel running in bhyve has > booted? I'm assuming "booted" in this sense to mean "PID 1 has = probably > started". I'm guessing that there probably isn't, but I thought I'd > better ask anyway. "Not booted" would mean something like "Is sitting > at the Grub prompt doing nothing=E2=80=9C. The abandoned FreeNAS Corral release did have guest additions that served as a simple health monitor for FreeBSD and Linux guests. They used virtio_console(4) and the last state of the project can be found here: https://github.com/freenas/freenas-vm-tools They compile cleanly on FreeBSD 12 and I even made a port that I submitted to iX Systems, but we all know what happened to Corral, unfortunately. Possibly they can serve as a starting point for a new project. And then there seems to be a newer version, again from FreeNAS, written in Go: https://github.com/freenas/bhyve-vm-goagent HTH, Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling --Apple-Mail=_976E1CCC-E48D-448E-8388-45E304997A64 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEzBAEBCgAdFiEEgzqrjO/mj9CSsTg2kG8u4u3aiVwFAlyGsm4ACgkQkG8u4u3a iVwQ5Af/VYevJImrzsbMch+UffUva+MUSILnZhXpH88TxiXaAz5FcIt14nVzRmey hLsoYRVGLEjfAWFRqNh5+2SiEas6xW2MqZtnRQP9TZ/CDtSVU1QozfblGRghA/5G T/u+XhENVctcg1RdRW62gwAGSAwjZyZ3TFW3PD7cVn9p+4K5rf1cLbVvZ23c7Szj wv+wOc9FQSPUedA+5okZwujOSgz0YDBJe54BEnqshjlxkd0XSJGGkD8g7UP+xlc8 +SRQ+mdEOm9rnh1g9758xn4BSa+iNFEiNrdsNjNpUaAJjDCVViEkGpByfqwwX4RL E9jJwqqyQYmcWKmQgqLhqu51tJz+xw== =E+Qg -----END PGP SIGNATURE----- --Apple-Mail=_976E1CCC-E48D-448E-8388-45E304997A64-- From owner-freebsd-virtualization@freebsd.org Mon Mar 11 19:12:37 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF65F152A7DF for ; Mon, 11 Mar 2019 19:12:37 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C2D076A72 for ; Mon, 11 Mar 2019 19:12:31 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x2BJCUp8076349; Mon, 11 Mar 2019 20:12:30 +0100 (CET) Received: from [217.29.46.2] (unassigned [217.29.46.2] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x2BJCSam081669; Mon, 11 Mar 2019 20:12:29 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: bhyve: Detecting that a guest kernel has booted From: "Patrick M. Hausen" In-Reply-To: <201903111809.x2BI974C010768@ gndrsh.dnsmgr.net> Date: Mon, 11 Mar 2019 20:12:13 +0100 Cc: Shawn Webb , freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <201903111809.x2BI974C010768@ gndrsh.dnsmgr.net> To: "Rodney W. Grimes" X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 9C2D076A72 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.59)[-0.591,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.43)[ipnet: 217.29.32.0/20(-1.19), asn: 16188(-0.96), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 19:12:37 -0000 Hi all, > Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes = : > I have no love for them either, but until we get our UEFI updated > it is a rather sad state of affairs. I routinely boot FreeBSD, Ubuntu and Windows 10 with UEFI, so I don=E2=80=99t see much that I would consider broken. Missing persistence of boot variables (correct term?) is the only thing that I know of - what else makes it a sad state, currently? Kind regards Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-virtualization@freebsd.org Mon Mar 11 20:35:43 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88F62152DC75 for ; Mon, 11 Mar 2019 20:35:43 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 92ABD82836 for ; Mon, 11 Mar 2019 20:35:42 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x2BKZeLU011551; Mon, 11 Mar 2019 13:35:40 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x2BKZd0a011550; Mon, 11 Mar 2019 13:35:39 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201903112035.x2BKZd0a011550@gndrsh.dnsmgr.net> Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: To: "Patrick M. Hausen" Date: Mon, 11 Mar 2019 13:35:39 -0700 (PDT) CC: "Rodney W. Grimes" , Shawn Webb , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 92ABD82836 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [1.63 / 15.00]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.50)[0.495,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: gndrsh.dnsmgr.net]; NEURAL_SPAM_LONG(0.64)[0.643,0]; NEURAL_HAM_SHORT(-0.40)[-0.405,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (0.05), ipnet: 69.59.192.0/19(0.03), asn: 13868(0.01), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 20:35:43 -0000 > Hi all, > > > Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes : > > I have no love for them either, but until we get our UEFI updated > > it is a rather sad state of affairs. > > I routinely boot FreeBSD, Ubuntu and Windows 10 with > UEFI, so I don?t see much that I would consider broken. All 64 bit oses, afaik we can not boot anything 32 bit with our current uefi, which is: This branch is 8930 commits behind tianocore:master. And depends on an external git belonging to Peter Grehan, who is now gone from the project. > > Missing persistence of boot variables (correct term?) is > the only thing that I know of - what else makes it a > sad state, currently? Our GOP code frequently blows up vnc clients is another issue, not sure if that is in the uefi stuff, or if we just have bad vnc layered on top of it. It throws recs that are out of bounds. The embeded PXE code is also rather dated. > Patrick -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Mar 11 21:18:08 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E487A152F05A for ; Mon, 11 Mar 2019 21:18:07 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 1BC2E8439F for ; Mon, 11 Mar 2019 21:18:06 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 120072091EE0 for ; Tue, 12 Mar 2019 07:18:03 +1000 (AEST) Received: from iredmail.onthenet.com.au (localhost [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id F2BD820B49AD for ; Tue, 12 Mar 2019 07:18:02 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dUrBYL2xDIKr for ; Tue, 12 Mar 2019 07:18:02 +1000 (AEST) Received: from iredmail.onthenet.com.au (localhost [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id DBCFF20B49A3; Tue, 12 Mar 2019 07:18:01 +1000 (AEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 11 Mar 2019 14:18:01 -0700 From: grehan@freebsd.org To: "Rodney W. Grimes" Cc: "Patrick M. Hausen" , freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: <201903112035.x2BKZd0a011550@gndrsh.dnsmgr.net> References: <201903112035.x2BKZd0a011550@gndrsh.dnsmgr.net> Message-ID: X-Sender: grehan@freebsd.org User-Agent: OntheNet Business Mail X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=IsKqj43g c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=NTGMnVQrEZIA:10 a=s1G7sxBSAAAA:20 a=qxzLhAw0Pjr_8Zf-MRYA:9 a=CjuIK1q_8ugA:10 a=bAsVuuXFBG0A:10 a=VlZU0XKO32wA:10 wl=host:3 X-Rspamd-Queue-Id: 1BC2E8439F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 21:18:08 -0000 > This branch is 8930 commits behind tianocore:master. > And depends on an external git belonging to Peter Grehan, No, it isn't. https://github.com/freebsd/uefi-edk2 Under FreeBSD project control since Oct 16, 2015. later, Peter. From owner-freebsd-virtualization@freebsd.org Tue Mar 12 00:20:35 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 236911533808; Tue, 12 Mar 2019 00:20:35 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 491EA8B0FC; Tue, 12 Mar 2019 00:20:34 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x2C0KWav012470; Mon, 11 Mar 2019 17:20:32 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x2C0KVVN012469; Mon, 11 Mar 2019 17:20:31 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201903120020.x2C0KVVN012469@gndrsh.dnsmgr.net> Subject: Re: bhyve: Detecting that a guest kernel has booted In-Reply-To: To: grehan@freebsd.org Date: Mon, 11 Mar 2019 17:20:31 -0700 (PDT) CC: "Rodney W. Grimes" , "Patrick M. Hausen" , freebsd-virtualization@freebsd.org, owner-freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 491EA8B0FC X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [3.27 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.70)[0.704,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: gndrsh.dnsmgr.net]; NEURAL_SPAM_LONG(0.84)[0.839,0]; NEURAL_SPAM_MEDIUM(0.84)[0.837,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (0.05), ipnet: 69.59.192.0/19(0.03), asn: 13868(0.01), country: US(-0.07)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2019 00:20:35 -0000 > > This branch is 8930 commits behind tianocore:master. > > And depends on an external git belonging to Peter Grehan, > > No, it isn't. > > https://github.com/freebsd/uefi-edk2 > > Under FreeBSD project control since Oct 16, 2015. My appologies, the page does not clearly show who owns the git project and I wrongly assumed it was you. Though perhaps there is another edk2 some place that some people seem to have copies of? Is the source the same for the versions in people.freebsd.org/~grehan/? > later, > Peter. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Tue Mar 12 17:00:08 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2807F1536ACD for ; Tue, 12 Mar 2019 17:00:08 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B2507137C for ; Tue, 12 Mar 2019 17:00:05 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 05F2C22903 for ; Tue, 12 Mar 2019 13:00:04 -0400 (EDT) Received: from imap2 ([10.202.2.52]) by compute1.internal (MEProxy); Tue, 12 Mar 2019 13:00:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=nPw6uComkkuEhrjgq0B5xfVp2P1z5dUFZ5wAc4exk5o=; b=cwKJyGw5 mjTNo7fvZqx80uZumHz3XXb1jnf635ZDpXmtOCua1uRd3uDELQIIadwAn+AHWaXv Z9MOVpFxXubAIU1XJGS8aVXk3SFRcEfyVcjJRz7+CpvCK0XpQyMEXotq7Z/5N4Pt A6iDtlBm8gDlfjGBunFqea37hy+WxUrIpB8XPQSS4GQpIskdJYMHzN8NEuBXLg8J 6GDZrkz6/gZKrP/xSiznpgYAH3tuEnkGPcuxUxduKKLJH06+wrKWfMNJi5GTjohi WWo3jcc0wKWZGGB8Xo/HsYdBdOSknA37ABQV+zH9GFWWULB5ZbBv8TAruLus86bn jbm7IwBrG8zCsg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrgeekgdeljecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfkfffhffvufgtsehttdertderre dtnecuhfhrohhmpedflfhoshhhucfrrggvthiivghlfdcuoehjohhshhesthgtsghughdr ohhrgheqnecurfgrrhgrmhepmhgrihhlfhhrohhmpehjohhshhesthgtsghughdrohhrgh enucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 014F37C538; Tue, 12 Mar 2019 13:00:02 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.5-925-g644bf8c-fmstable-20190228v5 X-Me-Personality: 4359293 Message-Id: <0219ca84-a0b7-438d-b43c-28526d505d98@www.fastmail.com> Date: Tue, 12 Mar 2019 13:01:17 -0400 From: "Josh Paetzel" To: freebsd-virtualization@freebsd.org Subject: Azure disk performance issue Content-Type: text/plain X-Rspamd-Queue-Id: 0B2507137C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=cwKJyGw5 X-Spamd-Result: default: False [-6.23 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-3.40)[ip: (-8.88), ipnet: 66.111.4.0/24(-4.60), asn: 11403(-3.46), country: US(-0.07)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[tcbug.org]; DKIM_TRACE(0.00)[messagingengine.com:+]; MX_GOOD(-0.01)[in2.smtp.messagingengine.com,in1.smtp.messagingengine.com,in2.smtp.messagingengine.com,in1.smtp.messagingengine.com,in2.smtp.messagingengine.com,in1.smtp.messagingengine.com]; NEURAL_HAM_SHORT(-0.93)[-0.932,0]; R_SPF_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[26.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2019 17:00:08 -0000 I have an issue where a FreeBSD VHD in Azure performs well when reading, and it performs well when writing, but if you throw a mixed read/write workload at it the write performance drops to near zero and latency starts spiking to several hundred milliseconds. Interestingly enough read performance improves under these conditions. Looking for some troubleshooting help. -- Thanks, Josh Paetzel From owner-freebsd-virtualization@freebsd.org Wed Mar 13 00:41:36 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC78A15452B1 for ; Wed, 13 Mar 2019 00:41:36 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B5738E30E for ; Wed, 13 Mar 2019 00:41:35 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id E9830393D for ; Tue, 12 Mar 2019 20:41:33 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Tue, 12 Mar 2019 20:41:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=gX9McWph1Y6ZZOL/OrscKfND2Je 1WkGqVvew4/sEaDo=; b=gfAdvdH2O6Wo6mjMMKEm3GfcBj/wWFRUhyPLamMw4Rl rJxwrEIdF5iz9fSyEvuArh3VapogLvT4wBbm3ojcMEqaobc7dV6c6gGNcnDhGWyL tg2IiZBKYMhhulaGbul0+TO84sV60A0C7JrfsJ7JoVeed0bKx++iJmu419O9DkWt +b9xMSeqQgKg95tmTb9vNM33XlXpKJfvh6+3CYIqEjkFoHYsa/RFo2zFtMs6DJ8M kC9mHb4YFX6MFftJo9GFxKaaFgRQ14BBHKXayDyl7VZrDylM9BZeaN5SreW2n0l3 uBmFwpEyBCoGxpjJYVMazDiTzI9+jnxRGzAkrSk8t3A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=gX9McW ph1Y6ZZOL/OrscKfND2Je1WkGqVvew4/sEaDo=; b=1fsyp5zvpgn1FBGXNN7Ej1 Q7W7dTbwQxof9KMneN7VFjkkey+m5C3X5MyfgxfNPeId2Cqccte+NSacki4d9zqe ifcjful6wC4whLkR4fmnD8W28+8Y/2pASRe1obdnXiOm4xHk8vGi2gHziOHqBscc BrpAVg67hyVAa9MXaVpmyEJwGmJ2j/OKrBFBjFq/KtZfPNj+7xRMnz2s24tzhijd Ze1Asg5IEufYIU7jyEEduSF7nChYOUN25mpaDJE2w4Hjqn02eJhZA8p0TUVq57Sm VKW4giH0Rb4MHSirYYc/xg7dvfG1ZxGJt5A/Wa4kRn4aZCnMYWcvcMlJsGJfhM2w == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrgeelgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjfgesthdtre dttdervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmh grihhlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushht vghrufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id D96F110336 for ; Tue, 12 Mar 2019 20:41:32 -0400 (EDT) Date: Wed, 13 Mar 2019 00:41:30 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Subject: Re: Azure disk performance issue Message-ID: <20190313004128.GA33467@rpi3.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org References: <0219ca84-a0b7-438d-b43c-28526d505d98@www.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <0219ca84-a0b7-438d-b43c-28526d505d98@www.fastmail.com> User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 9B5738E30E X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=gfAdvdH2; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=1fsyp5zv; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.25 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-6.97 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.25]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; MX_GOOD(-0.01)[cached: in2-smtp.messagingengine.com]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.985,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; IP_SCORE(-3.37)[ip: (-8.89), ipnet: 64.147.123.0/24(-4.44), asn: 11403(-3.46), country: US(-0.07)]; RCVD_IN_DNSWL_LOW(-0.10)[25.123.147.64.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2019 00:41:36 -0000 On Tue, Mar 12, 2019 at 01:01:17PM -0400, Josh Paetzel wrote: > I have an issue where a FreeBSD VHD in Azure performs well when reading, > and it performs well when writing, but if you throw a mixed read/write workload > at it the write performance drops to near zero and latency starts spiking to > several hundred milliseconds. > > Interestingly enough read performance improves under these conditions. > > Looking for some troubleshooting help. I'm very new to freebsd on azure so unfortunately can't be much help at least initially, but I'd be interested to know what sort of loads you're putting on it, and what resources the VM has (I mean # of cpu, ram, size). Any particular software triggering the problem? I have had to manage php5/mysql on a system where the "disk" was NFS or something very similar and sometimes it'd thrash under not a particularly heavy load. But that was years ago on linux. Some software needs attention or a setting for something like flock or dotlock or similar when used in a non-baremetal situation. -- J. From owner-freebsd-virtualization@freebsd.org Thu Mar 14 16:53:32 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1AF01549534 for ; Thu, 14 Mar 2019 16:53:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 333D4889BF for ; Thu, 14 Mar 2019 16:53:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E666F1549533; Thu, 14 Mar 2019 16:53:31 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C183F1549532 for ; Thu, 14 Mar 2019 16:53:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59D02889BA for ; Thu, 14 Mar 2019 16:53:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 957A9B04A for ; Thu, 14 Mar 2019 16:53:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x2EGrUJh072227 for ; Thu, 14 Mar 2019 16:53:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x2EGrUDP072226 for virtualization@FreeBSD.org; Thu, 14 Mar 2019 16:53:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 231117] I/O lockups inside bhyve vms Date: Thu, 14 Mar 2019 16:53:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: roel@qsp.nl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Mar 2019 16:53:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231117 roel@qsp.nl changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |roel@qsp.nl --- Comment #18 from roel@qsp.nl --- Just had this occur again on a VM running under bhyve with 12.0-STABLE, che= cked out and compiled 6 days ago (r344917). VM host is running the exact same kernel. The modifications in zfs_znode.c are present, but we still had an i= ssue after the system has been running for a couple of days. VM has >4GB arc_max (so the workaround as described by Kristian doesn't work for us): vfs.zfs.arc_min: 903779840 vfs.zfs.arc_max: 7230238720 Hypervisor: vfs.zfs.arc_min: 8216929792 vfs.zfs.arc_max: 65735438336 Procstat -kk on the bhyve process: root@cloud02:/home/roel # procstat -kk 18178 PID TID COMM TDNAME KSTACK=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 18178 101261 bhyve mevent mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a kqueue_kevent+0= x297 kern_kevent+0xb5 kern_kevent_generic+0x70 sys_kevent+0x61 amd64_syscall+0x3= 4d fast_syscall_common+0x101=20 18178 101731 bhyve vtnet-2:0 tx mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101732 bhyve blk-3:0:0-0 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101733 bhyve blk-3:0:0-1 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101734 bhyve blk-3:0:0-2 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101735 bhyve blk-3:0:0-3 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101736 bhyve blk-3:0:0-4 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101737 bhyve blk-3:0:0-5 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101738 bhyve blk-3:0:0-6 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101739 bhyve blk-3:0:0-7 mi_switch+0xe2 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x23a umtxq_sleep+0x1= 33 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0x34d fast_syscall_common+0x101=20 18178 101740 bhyve vcpu 0 =20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20 18178 101741 bhyve vcpu 1 mi_switch+0xe2 sleepq_timedwait+0x2f msleep_spin_sbt+0x138 vm_run+0x502 vmmdev_ioctl+0xbed devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x161 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15d amd64_syscall+0x34d fast_syscall_common+0x= 101=20 18178 101742 bhyve vcpu 2 mi_switch+0xe2 sleepq_timedwait+0x2f msleep_spin_sbt+0x138 vm_run+0x502 vmmdev_ioctl+0xbed devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x161 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15d amd64_syscall+0x34d fast_syscall_common+0x= 101=20 18178 101743 bhyve vcpu 3 mi_switch+0xe2 sleepq_timedwait+0x2f msleep_spin_sbt+0x138 vm_run+0x502 vmmdev_ioctl+0xbed devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x161 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15d amd64_syscall+0x34d fast_syscall_common+0x= 101=20 18178 101744 bhyve vcpu 4 mi_switch+0xe2 sleepq_timedwait+0x2f msleep_spin_sbt+0x138 vm_run+0x502 vmmdev_ioctl+0xbed devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x161 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15d amd64_syscall+0x34d fast_syscall_common+0x= 101=20 18178 101745 bhyve vcpu 5 mi_switch+0xe2 sleepq_timedwait+0x2f msleep_spin_sbt+0x138 vm_run+0x502 vmmdev_ioctl+0xbed devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x161 devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15d amd64_syscall+0x34d fast_syscall_common+0x= 101 --=20 You are receiving this mail because: You are the assignee for the bug.=