Date: Tue, 18 May 2021 16:56:55 -0400 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: John Doherty <bsdlists@jld3.net> Cc: freebsd-virtualization@freebsd.org Subject: Re: increasing bhyve VM_MAXCPU Message-ID: <20210518205655.2nnjt4yfdt43atby@mutt-hbsd> In-Reply-To: <7C6FC30B-BADC-4D32-AA2B-5F00B5E874F6@jld3.net> References: <7C6FC30B-BADC-4D32-AA2B-5F00B5E874F6@jld3.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--wszted6l6snkae6g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via freebsd-virtuali= zation wrote: > I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD > 12.2-RELEASE. >=20 > As I understand it, this limit is defined as VM_MAXCPU in > /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two > years ago, it should be possible to increase it beyond 21[1]. >=20 > I began working on a machine with only eight CPU cores/threads, so rather > than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to > work fine. My procedure was: >=20 > edit /usr/src/sys/amd64/include/vmm.h > cd /usr/src/lib/libvmmapi ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install > cd /usr/src ; make buildkernel KERNCONF=3DJLD ; make installkernel > KERNCONF=3DJLD > reboot >=20 > where JLD is a kernel config file that differs from GENERIC only in the > "ident" line. >=20 > I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs > with a number of vCPUs less than or equal to the max booted fine and that > VMs with more didn't. >=20 > So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores= /28 > threads each, using the same procedure. I set VM_MAXCPU to 28 and configu= red > a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20 > and booting a VM with 18 vCPUs but that didn't work either. In all cases, > VMs with 16 or fewer vCPUs continued to boot fine. >=20 > All the tests were done with VMs created before doing any of this stuff. I > am using vm-bhyve to manage VMs. The VMs in these tests were running one = of > FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing. >=20 > So it seems like I'm overlooking something but I don't know what, or maybe > this just isn't going to work. Anybody have experience actually doing thi= s? >=20 > Thanks very much for any advice or further information. HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our 14-CURRENT/amd64 package building VM is set up with 30 (out of 36 available) vCPUs (`vmrun.sh -c 30`). We've completed one full package build (so, 31,000+ ports) without issue. The one package building VM we have on the system (so no other VMs) is running HardenedBSD 14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs. [0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30 Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --wszted6l6snkae6g Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmCkKhQACgkQ/y5nonf4 4frv4g//WYnnfpb3LRxuTxRgwqKWZ9EGAH8TdKXbCKlQat4kcww9P78dzWNiR4/T EGm2r2IXMzY+f+tqECr1wu8Wuy+mJUTDotTm7ZP1+TdKJQsulBEDIZU5Quf+E/0e UfGegvaDySDD1cJ+dGV7H5dF8RRejZEwlZfyDXVpilL2jMaF6sjFI9H++9GHwOUF tj8ppTVxw65dviAWiu5Vw1/VgWDBfgif2cMf2iS+P/rna5Pcjt6KQbKgJBfm6zBy TgLtlf3YzmhJ94OveBoGmtoDUs5SP6wATni0JK5OsJgSRiGMGGpnBUCTNe7SEOJy ipbltjLtDojsmUwCeaMKQ9C2hR6moyCS2D5LNlSqE/hfG6FoNmKEAzgfmwaffxgb F7LqPbktzdj6RHdcLSI2bniLIQHat4EZTZ+SQU6HIqunKT1A3b9lfXJ0du9+Qk3Z HqUahFc0PNV+P2RD+rdhFHS21qR4y92kuWTz7buH6qjx5KV46TqDry+MooBGVRRg yD5IZA4EjeP84jsdEZEETctTX/D+4f6Axve2CjFYusafwSGROoHmJElEKhnReWBk 0CbskProCspXDa6yi9G3nIV35HBXWXR2xFuepCmCKO/qKtdCFPy3aS3vAP4dFoSq RtJJs/ATWdHD1uE3bTvOnCVTKjs9Nq2wtiKrdryNP7ZMmzX4iVE= =U1bc -----END PGP SIGNATURE----- --wszted6l6snkae6g--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20210518205655.2nnjt4yfdt43atby>