From owner-freebsd-virtualization@freebsd.org Thu Sep 10 03:34:09 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9AB03E7AA5 for ; Thu, 10 Sep 2020 03:34:09 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn4Hg4mtcz4MJg; Thu, 10 Sep 2020 03:34:07 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.160] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 2c242fb7 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Sep 2020 03:34:00 +0000 (UTC) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Peter Grehan Cc: FreeBSD virtualization References: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> From: Pete Wright Message-ID: <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> Date: Wed, 9 Sep 2020 20:33:55 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Bn4Hg4mtcz4MJg X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-2.53 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nomadlogic.org]; NEURAL_HAM_LONG(-0.95)[-0.949]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received]; NEURAL_HAM_MEDIUM(-1.03)[-1.030]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.25)[-0.249]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 03:34:09 -0000 On 9/9/20 5:36 PM, Peter Grehan wrote: > Hi Pete, > >>    VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr >  That's just a warning: bhyve will attempt to enable VT-x on module > load, and will only fail if BIOS has locked the disable. > >  Do you know if the vmm module fails to load ? (there will be an error > in syslog "vmx_init: VMX operation disabled by BIOS") thanks for getting back to me Peter.  The vmm.ko does load: $ kldstat|grep vmm 19    1 0xffffffff82ce1000   537420 vmm.ko $ Sep 10 03:25:01 bells kernel: vmx_init: processor does not support desired basic capabilities Sep 10 03:25:01 bells kernel: module_register_init: MOD_LOAD (vmm, 0xffffffff82ce4d20, 0) error 22 Sep 10 03:25:01 bells kernel: driver bug: Unable to set devclass (class: fdc devname: (unknown)) I also received this input from the ISP, they are running this kernel:  4.15.0-47-generic #50~16.04.1-Ubuntu SMP > >  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168 >   ... requires posted interrupts to be disabled. > reading through that ticket i set this in loader.conf: hw.vmm.vmx.use_apic_vid=0 and verified via sysctl: $ sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 $ since you mention this is a possible bug with KVM is there some useful information i can relay to my ISP (they are a BSD friendly vendor) that they can try on their end as a workaround, or perhaps something they ping the linux/kvm community about? thanks in advance! -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA