From owner-freebsd-virtualization@FreeBSD.ORG Tue Aug 6 23:10:41 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B22BCF13 for ; Tue, 6 Aug 2013 23:10:41 +0000 (UTC) (envelope-from mailinglists@debank.tv) Received: from smtpout100.onlinespamfilter.nl (smtpout100.onlinespamfilter.nl [IPv6:2001:4cb8:1:1620:217:21:240:168]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 385BD2A1E for ; Tue, 6 Aug 2013 23:10:41 +0000 (UTC) Received: from smtp.onlinespamfilter.nl (localhost [127.0.0.1]) by smtp.onlinespamfilter.nl (Postfix) with ESMTP id 4D51B207C5 for ; Wed, 7 Aug 2013 01:12:41 +0200 (CEST) Received: from smtp.debank.tv (145-158-ftth.on.nl [88.159.158.145]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.onlinespamfilter.nl (Postfix) with ESMTPS for ; Wed, 7 Aug 2013 01:12:41 +0200 (CEST) Received: from smtp.debank.tv (smtp.debank.tv [172.16.143.25]) by smtp.debank.tv (Postfix) with ESMTP id 789883239D8 for ; Wed, 7 Aug 2013 01:10:35 +0200 (CEST) Received: from debank.tv (roundcube.debank.tv [172.16.143.41]) by smtp.debank.tv (Postfix) with ESMTP id 3E8563239D3 for ; Wed, 7 Aug 2013 01:10:35 +0200 (CEST) MIME-Version: 1.0 Date: Wed, 07 Aug 2013 11:10:35 +1200 From: mailinglists@debank.tv To: freebsd-virtualization@freebsd.org Subject: Re: bhyve and virtualbox on the same host using EPT In-Reply-To: References: <1f2844d8e1f5f5f931811be351f19bd9@debank.tv> Message-ID: <52a20877a1dd07ab56e0a015a781f80f@debank.tv> X-Sender: mailinglists@debank.tv User-Agent: Roundcube Webmail/0.8.6 X-Virus-Scanned: ClamAV using ClamSMTP @ debank.tv X-OSF-Virus: CLEAN X-OSF-SUM: 009af07ee558f29af3b5b5261cc3c05c X-OSF-Info: Checked for spam and viruses Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Aug 2013 23:10:41 -0000 > On Tue, Aug 6, 2013 at 3:02 PM, wrote: > >> All, >> >> I'm trying to run a bhyve VM on a host that's already running Virtualbox, the problem I encounter is bhyve doesn't detect EPT capabilities but the Virtualbox logs indicate EPT is in use and functional. >> My question is; is it possible for two pieces of virtualization software to use EPT on the same host? >> >> See below for the various logs, happy to provide more info if needed. >> >> Bhyve generates the following error after the bootloader screen: >> >> ----------------------------------------------------- >> Booting... >> 4 vCPUs requested but only 1 available >> ----------------------------------------------------- >> >> A quick look in the source code indicates that this error means bhyve can't use EPT and falls back to using 1 CPU, now the machine in question does have EPT capabilites (it lists POPCNT in dmesg): >> >> ----------------------------------------------------- >> uname -a >> FreeBSD becks.debank.tv [1] 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r253788M: Tue Jul 30 12:02:57 NZST 2013 root@becks.debank.tv:/usr/obj/usr/src/sys/GENERIC amd64 >> ----------------------------------------------------- >> ----------------------------------------------------- >> from dmesg >> CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2664.82-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5 >> Features=0xbfebfbff >> Features2=0x98e3bd >> AMD Features=0x28100800 >> AMD Features2=0x1 >> TSC: P-state invariant, performance statistics >> real memory = 25769803776 (24576 MB) >> avail memory = 24060841984 (22946 MB) >> ----------------------------------------------------- >> ----------------------------------------------------- >> root@becks:~/bhyve # kldstat >> Id Refs Address Size Name >> 1 27 0xffffffff80200000 150e0d0 kernel >> 2 1 0xffffffff8170f000 224a98 zfs.ko >> 3 2 0xffffffff81934000 5c30 opensolaris.ko >> 4 1 0xffffffff8193a000 2d40 coretemp.ko >> 5 3 0xffffffff8193d000 51a68 vboxdrv.ko >> 6 2 0xffffffff81a12000 28c0 vboxnetflt.ko >> 7 2 0xffffffff81a15000 881d netgraph.ko >> 8 1 0xffffffff81a1e000 15ff ng_ether.ko >> 9 1 0xffffffff81a20000 3ec0 vboxnetadp.ko >> 10 1 0xffffffff81aba000 1037 cpuctl.ko >> 11 1 0xffffffff81abc000 1686ac vmm.ko >> ----------------------------------------------------- >> >> The Virtualbox logs below show EPT in use: >> ----------------------------------------------------- >> 00:00:00.419806 HWACCM: MSR_IA32_VMX_PROCBASED_CTLS2 = 7f00000000 >> 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VIRT_APIC >> 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_EPT >> 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_DESCRIPTOR_INSTR_EXIT >> 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_RDTSCP >> 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_X2APIC >> 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VPID >> 00:00:00.419810 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_WBINVD_EXIT >> ---- snip ------ >> 00:00:00.419916 HWACCM: 32-bit and 64-bit guests supported. >> 00:00:00.419917 HWACCM: VMX enabled! >> 00:00:00.419918 HWACCM: Enabled nested paging >> 00:00:00.419919 HWACCM: EPT root page = 00000000b6167000 >> 00:00:00.419922 HWACCM: Large page support enabled! >> ----------------------------------------------------- >> >> Thanks, >> Rob Evers >> >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization [2] >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" On 2013-08-07 10:33, Anish wrote: Each processor has its own EPT root pointer in VMCS. How many processors are there in system and how many are given to Virtualbox. error = vmcs_set_defaults(&vmx->vmcs[i], (u_long)vmx_longjmp, (u_long)&vmx->ctx[i], vtophys(vmx->pml4ept), pinbased_ctls, procbased_ctls, procbased_ctls2, exit_ctls, entry_ctls, vtophys(vmx->msr_bitmap), vpid); Anish, The machine has one physical CPU with 4 cores using HT giving 8 threads. I can't quickly find an option to Virtualbox to limit the amount of cpu's used but did try to disable Virtualbox (no rc.d scripts ran and disabled loading the kernel module) after a reboot of the machine, the problem persists. ----------------------------------------------------- root@becks:~/bhyve # kldstat Id Refs Address Size Name 1 17 0xffffffff80200000 150e0d0 kernel 2 1 0xffffffff8170f000 224a98 zfs.ko 3 2 0xffffffff81934000 5c30 opensolaris.ko 4 1 0xffffffff8193a000 2d40 coretemp.ko 5 1 0xffffffff81a12000 1037 cpuctl.ko 6 1 0xffffffff81a14000 1686ac vmm.ko ----------------------------------------------------- ----------------------------------------------------- root@becks:~/bhyve # sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 2048 fbsd_10 Launching virtual machine "fbsd_10" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@becks.debank.tv, Tue Jul 30 11:28:33 NZST 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0xd2f190 data=0x15de70+0x2baac0 syms=[0x8+0x14d648+0x8+0x1a9f58] -------------- snip ------------------ Booting... 4 vCPUs requested but only 1 available ----------------------------------------------------- Could there be a problem in the CPU capability discovery code in bhyve or is my machine not enabling EPT and Virtualbox wrongly reporting the EPT capabilities? Rob Evers Links: ------ [1] http://becks.debank.tv [2] http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization