Date: Fri, 17 Aug 2018 08:25:00 +0000 From: Matt Churchyard <matt.churchyard@userve.net> To: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>, Allan Jude <allanjude@FreeBSD.org> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@FreeBSD.org> Subject: RE: Checking bhyve supported features (sysctls) Message-ID: <c73da71b7ae342bd83175faff0f105cf@SERVER.ad.usd-group.com> In-Reply-To: <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net> References: <BC22EE63-357B-47F5-9121-A73B59633FE9@FreeBSD.org> <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
-----Original Message-----
From: owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualizatio=
n@freebsd.org> On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude <allanjude@FreeBSD.org>
Cc: Matt Churchyard <matt.churchyard@userve.net>; freebsd-virtualization@fr=
eebsd.org
Subject: Re: Checking bhyve supported features (sysctls)
> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <freebsd-rwg@=
pdx.rh.CN85.dnsmgr.net> wrote:
> >>=20
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >>=20
> >> > > Hello,
> >> > >=20
> >> > > I'm looking for better ways to check for bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> >=20
> >> > >Yes, it would be very good to remove that, as it usually tries=20
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not=20
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> >=20
> >> >=20
> >> > >=20
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > >=20
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > >=20
> >> > > Will these be available on both Intel & AMD processors as a way=20
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > >=20
> >> > > I also see the below sysctl related to iommu.
> >> > >=20
> >> > > hw.vmm.iommu.initialized
> >> > >=20
> >> > > Again, will this be set to 1 as soon as the module is loaded if=20
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to=20
> >> > > determine support on any processor.
> >> >=20
> >> > >Probalby the safest way for a shell script to decide if bhyve is=20
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> >=20
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >>=20
> >> I'll investigate that, I was pretty sure that you should get this=20
> >> as soon as the vmm.ko module is finished initialzing, but you might=20
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >>=20
> >> >=20
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> >=20
> >> > It would be nice if sysctls were better documented.
> >>=20
> >> Agreed.
> >>=20
> >> > If vmx.initialized is set once vmm has successfully loaded, I=20
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not=20
> >Intel specific). This entry definitely exists and is set to 0 if you=20
> >load the module on a non-supported system, and set to 1 as soon as=20
> >vmm loads on my Intel test system.
> >>=20
> >> Given its undocumented status you would be relying on an=20
> >> undocumented feature that could change in either name or behavior,=20
> >> and that is not desirable.
> >>=20
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve. They do not=20
> >actually try to decide if vmm is supported or not, they simply=20
> >process the error from a vm_create() or vm_open() call and exit with=20
> >an error code if they can not handle it (some of the code can handle=20
> >a vm_create failure if infact we are trying to create a vm that=20
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may=20
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has=20
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the=20
> >supported functions do not exist?
>=20
> I think the question vm-bhyve wants to answer is: does the CPU have=20
> the required features to run a multicore VM.
>Why does it need to know that? If it tries to start a multicore/thread VM=
and the system can not support it an error is returned and the bhyve comma=
nd fails.
>Actually determining that specific issue is non-trivial iirc as some vmm s=
upported CPU's can only run a single VM with a single thread in that VM (ea=
rly core cpu's).
>=20
> These or similar sysctls do seem to be the correct way to communicate=20
> that support.
>I do not believe any of those sysctl's communicate that your on a broken c=
pu or to what extent you can run vm's with multiple threads.
So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not=
a valid way to determine if the host has unrestricted guest support (requi=
red for non-freebsd or multicore freebsd guests, and as you say missing fro=
m some early VT-x capable processors)?
>I went and looked at why vm-bhyve is groveling around in /var/run/dmesg.bo=
ot and found that it is simply trying to determine if the host CPU is vmm c=
apable,
>specifically:
>util::check_bhyve_support(){
>...
>These checks are already built into the kernel.
>This can all go in the bit bucket, if you try to start a VM on an unsuppor=
ted system an error is returned, recoding this in shell is just setting you=
rself up for "future" bugs.
The kernel knows what features are supported but does not expose these, so =
all I can do is similar to libvirt and run bhyve with different options to =
see what errors pop up.
I think I'll just remove all checking for now and let users discover the is=
sue for themselves if bhyve won't run. Hopefully the vmx.initialized / cap.=
* sysctls will at some point become a defined way of seeing if vmm is ready=
/ testing for vmm features, as apparently these serve no purpose at the mo=
ment.
Matt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c73da71b7ae342bd83175faff0f105cf>