From owner-freebsd-virtualization@freebsd.org Wed May 19 01:23:00 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A16D5634331 for ; Wed, 19 May 2021 01:23:00 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FlFVX00xDz3MJG for ; Wed, 19 May 2021 01:22:59 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 14J1MwB7001153; Tue, 18 May 2021 18:22:58 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 14J1Mwfa001152; Tue, 18 May 2021 18:22:58 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202105190122.14J1Mwfa001152@gndrsh.dnsmgr.net> Subject: Re: increasing bhyve VM_MAXCPU In-Reply-To: <20210518205655.2nnjt4yfdt43atby@mutt-hbsd> To: Shawn Webb Date: Tue, 18 May 2021 18:22:58 -0700 (PDT) CC: John Doherty , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4FlFVX00xDz3MJG X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [1.80 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.973]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dnsmgr.net]; RBL_DBL_DONT_QUERY_IPS(0.00)[69.59.192.140:from]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.98)[0.976]; SPAMHAUS_ZRD(0.00)[69.59.192.140:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.90)[0.902]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 May 2021 01:23:00 -0000 -- Start of PGP signed section. > On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via freebsd-virtualization wrote: > > I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD > > 12.2-RELEASE. > > > > As I understand it, this limit is defined as VM_MAXCPU in > > /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two > > years ago, it should be possible to increase it beyond 21[1]. > > > > I began working on a machine with only eight CPU cores/threads, so rather > > than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to > > work fine. My procedure was: > > > > edit /usr/src/sys/amd64/include/vmm.h > > cd /usr/src/lib/libvmmapi ; make clean ; make ; make install > > cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install > > cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install > > cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install > > cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel > > KERNCONF=JLD > > reboot > > > > where JLD is a kernel config file that differs from GENERIC only in the > > "ident" line. > > > > I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs > > with a number of vCPUs less than or equal to the max booted fine and that > > VMs with more didn't. > > > > So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores/28 > > threads each, using the same procedure. I set VM_MAXCPU to 28 and configured > > a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20 > > and booting a VM with 18 vCPUs but that didn't work either. In all cases, > > VMs with 16 or fewer vCPUs continued to boot fine. > > > > All the tests were done with VMs created before doing any of this stuff. I > > am using vm-bhyve to manage VMs. The VMs in these tests were running one of > > FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing. > > > > So it seems like I'm overlooking something but I don't know what, or maybe > > this just isn't going to work. Anybody have experience actually doing this? > > > > Thanks very much for any advice or further information. > > HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our > 14-CURRENT/amd64 package building VM is set up with 30 (out of 36 > available) vCPUs (`vmrun.sh -c 30`). We've completed one full package > build (so, 31,000+ ports) without issue. The one package building VM > we have on the system (so no other VMs) is running HardenedBSD > 14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs. > > [0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30 When I made the changes to fix the brokeness at 21 vCPU we (Michael Dexter and myself) did testing into the 200's of vcpu's. Note that many guests have problems at such large core counts, but bhyve itself behaved ok, but some what sluggish as iirc the testing was done on a 32 core system. > Thanks, > > -- > Shawn Webb > Cofounder / Security Engineer > HardenedBSD > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc -- End of PGP section, PGP failed! -- Rod Grimes rgrimes@freebsd.org