Date: Wed, 12 Nov 2014 18:46:36 -0500 From: Allan Jude <allanjude@freebsd.org> To: freebsd-virtualization@freebsd.org Subject: Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance? Message-ID: <5463F15C.6070103@freebsd.org> In-Reply-To: <CAFgRE9Hc9q6Hrv25-g50jfXn3Havy186DQMcW9oV8%2B1=BxN=pA@mail.gmail.com> References: <8e39523570b667d223091adc3791c9e2@openmailbox.org> <5463D746.4030803@freebsd.org> <49cc2695179830c899aef87b7a8b72d8@openmailbox.org> <CAFgRE9Hc9q6Hrv25-g50jfXn3Havy186DQMcW9oV8%2B1=BxN=pA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --PRKRNQXt4xFtxCCkBG9Dd8TkXAFHG1FXR Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2014-11-12 17:31, Neel Natu wrote: > Hi Tinker, >=20 > On Wed, Nov 12, 2014 at 2:08 PM, Tinker <tinkr@openmailbox.org> wrote: >> On 2014-11-12 23:55, Allan Jude wrote: >>> >>> On 2014-11-12 16:39, tinkr@openmailbox.org wrote: >>>> >>>> Hi! >>>> >>>> In order justify giving energy to BHyVe, I need to know if it's >>>> "future-proof" in that the 16 vCPU limit can be increased? >>>> >>>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in = some >>>> way, by configuration, patch, etc. (and if you want to, why this lim= it >>>> is in place by default today). >>>> >>>> Thanks!! >>>> Tinker >>>> >>>> _______________________________________________ >>>> freebsd-virtualization@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>>> To unsubscribe, send any mail to >>>> "freebsd-virtualization-unsubscribe@freebsd.org" >>> >>> >>> You can increase the limit by editing sys/amd64/include/vmm.h >>> >>> #define VM_MAXCPU 16 >>> >>> From what I've been told, things scale badly above 24 CPUs. They plan= to >>> solve this issue, but have not yet. If you system has enough cores to= >>> support using more than 16 per VM, you can modify the file and recomp= ile >>> the kernel and use as many CPUs as you want, but not much testing has= >>> happened with bigger numbers. >> >> >> Dear Allan, >> >> Thank you very much for responding. >> >> Aha, great! >> >> >> Only for completeness, do you have any particular idea about >> * When the scaling above 24 vCPU:s will be optimized, like approx how= many >> years away is it (like 1 or more than 1)?, and >=20 > bhyve allocates memory for the maximum number of vcpus when the > virtual machine is created. This is simple to implement and also fits > in nicely with bhyve's loader model where the guest loader > (bhyveload/grub-bhyve) is separate from bhyve. >=20 > The downside is that if VM_MAXCPU is a large number then there is a > large memory penalty for virtual machines that only use 1 or 2 vcpus. > Hence the desire to cap it at a smallish number. >=20 > There are a few ways to deal with this: > - patch the code to change VM_MAXCPU (this is what you need to do today= ) > - allow the maximum vcpus to be changed via a tunable (this can be > done in short order) > - the limit can go away when bhyve moves to a single process model > because we'll know the number of vcpus at VM creation time. >=20 >> * What the technological reason for the scaling is, is it that someho= w the >> BHyVe instances on the different cores need to inter-communicate, for >> instance that all disk and network IO is done via one single core curr= ently? >> >=20 > Actually, the performance depends more on the workload and the guest > OS so you should definitely try how it goes for your application. > We'll be happy to help fix any issues that come up. >=20 > best > Neel >=20 >> >> In all cases, your response is great news, as your baseline answer tha= t it's >> doable and only a question of optimization and tweaking of present >> functionality - >> >> Thanks again! >> >> Tinker >> Right, I forgot that at MeetBSD it was decided that making the max cpus a sysctl was a good stop-gap measure to hold us over until the redesign of bhyveload. --=20 Allan Jude --PRKRNQXt4xFtxCCkBG9Dd8TkXAFHG1FXR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJUY/FlAAoJEJrBFpNRJZKfhAoP/i1Ed8s13RRjI6AcLmsZYDy1 nCk7Sz6dk1OwmZoBJqO5jEGUZbRpmBSv6Df0MGsdKvPfog5t0D6MTo91T2z0e1ma xadZw4EwiIbm7ZeLXRNjOyhvYOrqga6T4lKENxJ7QvPKVy90nMi9j8xIb8OAVHO+ dGKQrFrhsYBnhUQp63y7pwTc2DM5RwRysEiXVtDFx9Ofdj7Eochg2PTCdiwRzUe9 dERdzMf89xdqVd7+4fonTvVSOOZ2PG8JzoJri2f05x7CPkl8KerCx8l//ARxDOHk Pu9dBQIoOlgLvF4tRDzBL1bwqV073Lx8ZsPgWpolT/9BjqAVXWiof4AiNXQ+vAMd M5+YvncLEQYgP4h4pvppx0akZoW1nIxE7M0vkqaVOyx1AntTFvJFmitEOlO1ZILX 2qwATN1JlZwm0DK2zX9XV7t4xEdYo09ONU4T6MSVmP3P73zaXOcAliQBk8NfSVBw qOScZcPLGnpaAMHbDzk+yg23HX43ZmKKqJJkqAE/gNJyiC9tMRmlFk59CxAhaM/2 FzrRCoj0sOMGgWe+JvkOTituv3WECdDiwkfGRZyDjQYBuiDFfwoGB1icysiUo9+O xyQv1vmM9Xfe+RO68dopi5P8b8zgVT/b29HQ3OlUmO+jQ0gxQzGusRlY4ZVgOLSp TqsNefle7dcJ9rCJ8ySS =jYFN -----END PGP SIGNATURE----- --PRKRNQXt4xFtxCCkBG9Dd8TkXAFHG1FXR--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5463F15C.6070103>