From owner-freebsd-virtualization@freebsd.org Tue May 18 20:51:40 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8CE776571ED for ; Tue, 18 May 2021 20:51:40 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fl7TR58Q0z3GQv for ; Tue, 18 May 2021 20:51:39 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id 694CE415B5 for ; Tue, 18 May 2021 14:51:33 -0600 (MDT) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 4Q882j1bsTZQ for ; Tue, 18 May 2021 14:51:32 -0600 (MDT) Received: from [172.21.35.202] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id 00C9940573 for ; Tue, 18 May 2021 14:51:31 -0600 (MDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net 00C9940573 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net 00C9940573 From: "John Doherty" To: freebsd-virtualization@freebsd.org Subject: increasing bhyve VM_MAXCPU Date: Tue, 18 May 2021 14:51:22 -0600 X-Mailer: MailMate (1.13.2r5673) Message-ID: <7C6FC30B-BADC-4D32-AA2B-5F00B5E874F6@jld3.net> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Rspamd-Queue-Id: 4Fl7TR58Q0z3GQv X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[45.55.236.93:from]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[45.55.236.93:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[jld3.net:+]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RECEIVED_SPAMHAUS_PBL(0.00)[24.9.144.115:received] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 May 2021 20:51:40 -0000 I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD 12.2-RELEASE. As I understand it, this limit is defined as VM_MAXCPU in /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two years ago, it should be possible to increase it beyond 21[1]. I began working on a machine with only eight CPU cores/threads, so rather than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to work fine. My procedure was: edit /usr/src/sys/amd64/include/vmm.h cd /usr/src/lib/libvmmapi ; make clean ; make ; make install cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel KERNCONF=JLD reboot where JLD is a kernel config file that differs from GENERIC only in the "ident" line. I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs with a number of vCPUs less than or equal to the max booted fine and that VMs with more didn't. So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores/28 threads each, using the same procedure. I set VM_MAXCPU to 28 and configured a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20 and booting a VM with 18 vCPUs but that didn't work either. In all cases, VMs with 16 or fewer vCPUs continued to boot fine. All the tests were done with VMs created before doing any of this stuff. I am using vm-bhyve to manage VMs. The VMs in these tests were running one of FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing. So it seems like I'm overlooking something but I don't know what, or maybe this just isn't going to work. Anybody have experience actually doing this? Thanks very much for any advice or further information. [1]