From owner-freebsd-virtualization@freebsd.org Wed Sep 9 14:31:27 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A91D63D223D for ; Wed, 9 Sep 2020 14:31:27 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmkwZ172Pz4KZL for ; Wed, 9 Sep 2020 14:31:25 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.159] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id bbd1fd39 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 9 Sep 2020 14:31:24 +0000 (UTC) To: FreeBSD virtualization From: Pete Wright Subject: Nested Virtualization Under KVM Hypervisor Message-ID: Date: Wed, 9 Sep 2020 07:31:23 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BmkwZ172Pz4KZL X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-1.90 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_SPAM_SHORT(0.05)[0.046]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.75)[-0.750]; DMARC_NA(0.00)[nomadlogic.org]; TO_DN_ALL(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.894]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 14:31:27 -0000 Hey there, I have a host that is running via hardware virtualization under KVM and am having issues with virtualization.  My provider investigated and ensured that the hypervisor was configured to pass through VT-x to guests, and reports that Linux guests are detecting VT-x and are able to run nested virtualization on their systems.  For FreeBSD this is not working - dmesg is reporting this: CPU: Intel Xeon E312xx (Sandy Bridge) (2600.16-MHz K8-class CPU)   Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a Stepping=1 Features=0x783fbff Features2=0x9fb82223   AMD Features=0x28100800   AMD Features2=0x1   XSAVE Features=0x1   VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Hypervisor: Origin = "KVMKVMKVM" real memory  = 34359738368 (32768 MB) avail memory = 33365536768 (31819 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: They did verify that VT-x pass through *is* enabled as mentioned above (they shared with me the config file snippet on their end to verify), and they did extensive testing on their end and were not able to get FreeBSD to detect VT-x as being enabled. I pinged them asking for more info on which Linux kernel their hypervisors are running to see if I can help debug this on my end.  I have two questions: 1) is anyone else doing nested virtualization under KVM hypervisors.  i am confident that this is possible, but am asking for confirmation on this :) 2) Are there any other requirements that FreeBSD requires to run Bhyve nested under KVM? Thanks in advance! -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA