From owner-freebsd-virtualization@freebsd.org Fri Aug 17 08:25:04 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33F0010867BB for ; Fri, 17 Aug 2018 08:25:04 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C52FD7225C; Fri, 17 Aug 2018 08:25:03 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 29EA38DD; Fri, 17 Aug 2018 09:25:01 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1534494301; bh=LawhPjb+exT33HXKCzjt2YhJ6KSe9MRMHUVnX0qqhpc=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=dc+KYB0VFfq6W2inIU9kx4MjZ8yqJ4uYIzaB91FX+/YOMXqmtnKqPs7sG0z72o9Ok Yo7RYq59Pw1b7+37lOOQi8NywrCOw0H2G0ZPf5qfrc72JxAFTPVtPzf9KABFT5ua8/ 0M9Km4J/Xh05fCED1B7yqzBTEvuUAvyLOQEkCw4Q= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Fri, 17 Aug 2018 09:25:00 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Fri, 17 Aug 2018 09:25:00 +0100 From: Matt Churchyard To: "Rodney W. Grimes" , Allan Jude CC: "freebsd-virtualization@freebsd.org" Subject: RE: Checking bhyve supported features (sysctls) Thread-Topic: Checking bhyve supported features (sysctls) Thread-Index: AdQ1Sj/JzUfq4S8/RYq4DRXDNWTckQAIYkuAAANnXRD///SbAIAABAyAgAAHIQCAAApWAP/++Rig Date: Fri, 17 Aug 2018 08:25:00 +0000 Message-ID: References: <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2018 08:25:04 -0000 -----Original Message----- From: owner-freebsd-virtualization@freebsd.org On Behalf Of Rodney W. Grimes Sent: 16 August 2018 18:31 To: Allan Jude Cc: Matt Churchyard ; freebsd-virtualization@fr= eebsd.org Subject: Re: Checking bhyve supported features (sysctls) > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" wrote: > >>=20 > >> Text manually wrapped to 80, any broken quoting is my fault - rwg > >>=20 > >> > > Hello, > >> > >=20 > >> > > I'm looking for better ways to check for bhyve support / > >available > >> > > features without trying to scan through dmesg output. > >> >=20 > >> > >Yes, it would be very good to remove that, as it usually tries=20 > >> > >to grep a non-existent file /var/run/dmesg.boot that is not=20 > >> > >created until after vm_bhyve has been called from > >/usr/local/etc/rc.d > >> > >when you have things set to autostartup >in /etc/rc.conf > >> >=20 > >> >=20 > >> > >=20 > >> > > I notice that the following 2 sysctl's appear to be set to 1 as > >soon > >> > > as the vmm module is loaded > >> > >=20 > >> > > hw.vmm.vmx.initialized: 1 > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1 > >> > >=20 > >> > > Will these be available on both Intel & AMD processors as a way=20 > >> > > to determine if the module has loaded successfully and can run > >guests? > >> > >=20 > >> > > I also see the below sysctl related to iommu. > >> > >=20 > >> > > hw.vmm.iommu.initialized > >> > >=20 > >> > > Again, will this be set to 1 as soon as the module is loaded if=20 > >> > > iommu is supported, or only when it is used? > >> > > There also seems to be a vmm.amdvi.enable sysctl. > >> > > Would both these need checking or is vmm.iommu enough to=20 > >> > > determine support on any processor. > >> >=20 > >> > >Probalby the safest way for a shell script to decide if bhyve is=20 > >> > >up and running is to stat /dev/vmm, if that exists then the > >modules > >> > >have loaded and initialized and bhyve should be ready to process > >guests. > >> >=20 > >> > Hmm, I don't get /dev/vmm unless I actually have running guests. > >>=20 > >> I'll investigate that, I was pretty sure that you should get this=20 > >> as soon as the vmm.ko module is finished initialzing, but you might=20 > >> be right in that it takes a first vm to cause its creation. > >> Confirmed, /dev/vmm does not exist until the first vm is created. > >>=20 > >> >=20 > >> > >sysctl's mentiond above would be a poor way to make this > >determination. > >> >=20 > >> > It would be nice if sysctls were better documented. > >>=20 > >> Agreed. > >>=20 > >> > If vmx.initialized is set once vmm has successfully loaded, I=20 > >> > can't > >see a better way of checking for bhyve support (assuming it's not=20 > >Intel specific). This entry definitely exists and is set to 0 if you=20 > >load the module on a non-supported system, and set to 1 as soon as=20 > >vmm loads on my Intel test system. > >>=20 > >> Given its undocumented status you would be relying on an=20 > >> undocumented feature that could change in either name or behavior,=20 > >> and that is not desirable. > >>=20 > >> Let me see if I can come up with something else. > > > >I looked at the code for bhyvectl, bhyveload and byhve. They do not=20 > >actually try to decide if vmm is supported or not, they simply=20 > >process the error from a vm_create() or vm_open() call and exit with=20 > >an error code if they can not handle it (some of the code can handle=20 > >a vm_create failure if infact we are trying to create a vm that=20 > >already exists). > > > >If you want to maintain full compatibility a similiar stratergy may=20 > >be in order. > > > >Why is it that vm-bhyve specifically needs to know if the kernel has=20 > >vmm support or not? > >Cant it just be written to handle the errors returned if the=20 > >supported functions do not exist? >=20 > I think the question vm-bhyve wants to answer is: does the CPU have=20 > the required features to run a multicore VM. >Why does it need to know that? If it tries to start a multicore/thread VM= and the system can not support it an error is returned and the bhyve comma= nd fails. >Actually determining that specific issue is non-trivial iirc as some vmm s= upported CPU's can only run a single VM with a single thread in that VM (ea= rly core cpu's). >=20 > These or similar sysctls do seem to be the correct way to communicate=20 > that support. >I do not believe any of those sysctl's communicate that your on a broken c= pu or to what extent you can run vm's with multiple threads. So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not= a valid way to determine if the host has unrestricted guest support (requi= red for non-freebsd or multicore freebsd guests, and as you say missing fro= m some early VT-x capable processors)? >I went and looked at why vm-bhyve is groveling around in /var/run/dmesg.bo= ot and found that it is simply trying to determine if the host CPU is vmm c= apable, >specifically: >util::check_bhyve_support(){ >... >These checks are already built into the kernel. >This can all go in the bit bucket, if you try to start a VM on an unsuppor= ted system an error is returned, recoding this in shell is just setting you= rself up for "future" bugs. The kernel knows what features are supported but does not expose these, so = all I can do is similar to libvirt and run bhyve with different options to = see what errors pop up. I think I'll just remove all checking for now and let users discover the is= sue for themselves if bhyve won't run. Hopefully the vmx.initialized / cap.= * sysctls will at some point become a defined way of seeing if vmm is ready= / testing for vmm features, as apparently these serve no purpose at the mo= ment. Matt