Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 17 Aug 2018 17:14:27 +0800
From:      Marcelo Araujo <araujobsdport@gmail.com>
To:        Matt Churchyard <matt.churchyard@userve.net>
Cc:        Allan Jude <allanjude@freebsd.org>,  "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>
Subject:   Re: Checking bhyve supported features (sysctls)
Message-ID:  <CAOfEmZiJXUAyMmKCwXkP-aifhRz_eFotYFt5vuCpNRuEwv4P2g@mail.gmail.com>
In-Reply-To: <8fad62d974804195bf4518a88b7c53a8@SERVER.ad.usd-group.com>
References:  <BC22EE63-357B-47F5-9121-A73B59633FE9@FreeBSD.org> <201808161730.w7GHUaWv054788@pdx.rh.CN85.dnsmgr.net> <c73da71b7ae342bd83175faff0f105cf@SERVER.ad.usd-group.com> <CAOfEmZjSbnYaNEj%2BG=mVsdeeWb_GGYdS9xXNehqSRXRWAjLhdw@mail.gmail.com> <8fad62d974804195bf4518a88b7c53a8@SERVER.ad.usd-group.com>

next in thread | previous in thread | raw e-mail | index | archive | help
2018-08-17 16:54 GMT+08:00 Matt Churchyard <matt.churchyard@userve.net>:

>
>
>
>
> 2018-08-17 16:25 GMT+08:00 Matt Churchyard <matt.churchyard@userve.net>:
>
> -----Original Message-----
> From: owner-freebsd-virtualization@freebsd.org <
> owner-freebsd-virtualization@freebsd.org> On Behalf Of Rodney W. Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude <allanjude@FreeBSD.org>
> Cc: Matt Churchyard <matt.churchyard@userve.net>; freebsd-virtualization@
> freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
>
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <
> freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote:
> > >>
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >>
> > >> > > Hello,
> > >> > >
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> >
> > >> > >Yes, it would be very good to remove that, as it usually tries
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> >
> > >> >
> > >> > >
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > >
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > >
> > >> > > Will these be available on both Intel & AMD processors as a way
> > >> > > to determine if the module has loaded successfully and can run
> > >guests?
> > >> > >
> > >> > > I also see the below sysctl related to iommu.
> > >> > >
> > >> > > hw.vmm.iommu.initialized
> > >> > >
> > >> > > Again, will this be set to 1 as soon as the module is loaded if
> > >> > > iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to
> > >> > > determine support on any processor.
> > >> >
> > >> > >Probalby the safest way for a shell script to decide if bhyve is
> > >> > >up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to process
> > >guests.
> > >> >
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >>
> > >> I'll investigate that, I was pretty sure that you should get this
> > >> as soon as the vmm.ko module is finished initialzing, but you might
> > >> be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >>
> > >> >
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> >
> > >> > It would be nice if sysctls were better documented.
> > >>
> > >> Agreed.
> > >>
> > >> > If vmx.initialized is set once vmm has successfully loaded, I
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not
> > >Intel specific). This entry definitely exists and is set to 0 if you
> > >load the module on a non-supported system, and set to 1 as soon as
> > >vmm loads on my Intel test system.
> > >>
> > >> Given its undocumented status you would be relying on an
> > >> undocumented feature that could change in either name or behavior,
> > >> and that is not desirable.
> > >>
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do not
> > >actually try to decide if vmm is supported or not, they simply
> > >process the error from a vm_create() or vm_open() call and exit with
> > >an error code if they can not handle it (some of the code can handle
> > >a vm_create failure if infact we are trying to create a vm that
> > >already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel has
> > >vmm support or not?
> > >Cant it just be written to handle the errors returned if the
> > >supported functions do not exist?
> >
> > I think the question vm-bhyve wants to answer is: does the CPU have
> > the required features to run a multicore VM.
>
> >Why does it need to know that?  If it tries to start a multicore/thread
> VM and the system can not support it an error is returned and the bhyve
> command fails.
>
> >Actually determining that specific issue is non-trivial iirc as some vmm
> supported CPU's can only run a single VM with a single thread in that VM
> (early core cpu's).
>
> >
> > These or similar sysctls do seem to be the correct way to communicate
> > that support.
>
> >I do not believe any of those sysctl's communicate that your on a broken
> cpu or to what extent you can run vm's with multiple threads.
>
> So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is
> not a valid way to determine if the host has unrestricted guest support
> (required for non-freebsd or multicore freebsd guests, and as you say
> missing from some early VT-x capable processors)?
>
> >I went and looked at why vm-bhyve is groveling around in
> /var/run/dmesg.boot and found that it is simply trying to determine if th=
e
> host CPU is vmm capable,
> >specifically:
> >util::check_bhyve_support(){
> >...
>
> >These checks are already built into the kernel.
> >This can all go in the bit bucket, if you try to start a VM on an
> unsupported system an error is returned, recoding this in shell is just
> setting yourself up for "future" bugs.
>
> The kernel knows what features are supported but does not expose these, s=
o
> all I can do is similar to libvirt and run bhyve with different options t=
o
> see what errors pop up.
> I think I'll just remove all checking for now and let users discover the
> issue for themselves if bhyve won't run. Hopefully the vmx.initialized /
> cap.* sysctls will at some point become a defined way of seeing if vmm is
> ready / testing for vmm features, as apparently these serve no purpose at
> the moment.
>
> Matt
>
> _______________________________________________
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-
> unsubscribe@freebsd.org"
>
>
>
> >This is something that is in my long to-do list, I will try to get back
> to this in couple weeks.
>
> >I think the way how you are dealing with it nowadays is the best way to
> try to discover the CPU capabilities, better in this way than let the use=
rs
> blind.
>
> >Best,
>
> Thanks Marcelo,
>
>
>
> It=E2=80=99s not exactly critical. I just saw another project using
> vmx.initialized and it seemed a much cleaner way to determine support by
> seeing if vmm loads rather than probing log files. I asked here expecting
> someone to just say =E2=80=9Cyes if that=E2=80=99s 1 then the host will r=
un guests=E2=80=9D, or =E2=80=9Cno
> you can=E2=80=99t tell if guests will run just by that sysctl=E2=80=9D.
>
>
>
> Matt
>

Could you share with me what project is using vmx.initialized?


Best,

--=20

--=20
Marcelo Araujo            (__)araujo@FreeBSD.org
\\\'',)http://www.FreeBSD.org <http://www.freebsd.org/>;   \/  \ ^
Power To Server.         .\. /_)



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOfEmZiJXUAyMmKCwXkP-aifhRz_eFotYFt5vuCpNRuEwv4P2g>