Date: Sat, 17 Sep 2016 20:30:57 -0400 From: Allan Jude <allanjude@freebsd.org> To: freebsd-virtualization@freebsd.org Subject: Re: High vCPU Counts in bhyve Message-ID: <4ede10b9-95b2-f5ba-522e-de08a3937068@freebsd.org> In-Reply-To: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> References: <CAE7bWBoH0JWR5m5dFpEW=fg8XUaKjPWak7yTra_PWnJ8GCQLbA@mail.gmail.com> <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe Content-Type: multipart/mixed; boundary="VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: freebsd-virtualization@freebsd.org Message-ID: <4ede10b9-95b2-f5ba-522e-de08a3937068@freebsd.org> Subject: Re: High vCPU Counts in bhyve References: <CAE7bWBoH0JWR5m5dFpEW=fg8XUaKjPWak7yTra_PWnJ8GCQLbA@mail.gmail.com> <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> In-Reply-To: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> --VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-09-17 19:55, Peter Grehan wrote: > Hi Trent, >=20 >> If anyone has any hints on how to get more information from this or id= eas >> on the apparent APIC error would be greatly appreciated. >=20 > Thanks for the boot logs. The bhyve bug can be seen from: >=20 > ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, > should be 0x0A (20160527/tbprint-229) > [7/1843] > MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ= 9 >=20 > .. which is pointing to the MADT table being overwritten. There is onl= y > 256 bytes allocated for this table when it is being created: >=20 > acpi.c > * MADT -> 0xf2500 (depends on #CPUs) > * FADT -> 0xf2600 (268 bytes) > ... > #define MADT_OFFSET 0x100 > #define FADT_OFFSET 0x200 >=20 > The MADT* for bhyve has a 44-byte fixed header, followed by a array of= > 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of th= e > table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source > Override' entries, and a 6-byte 'Local APIC NMI' entry. >=20 > Looking at the max #CPUs that can fit into 256 bytes: >=20 > 256 =3D 44 + N*8 + 12 + 2*10 + 6, which gives N =3D 21. The fact that i= t > worked for slightly larger values is probably due to the table entries > at the end being ignored, until eventually the I/O APIC table entry was= > corrupted by the FADT overwrite. >=20 > A quick fix to get more vCPUs is to bump the addresses of the tables in= > acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. >=20 > +#define FADT_OFFSET 0x700 > +#define HPET_OFFSET 0x840 > +#define MCFG_OFFSET 0x880 > +#define FACS_OFFSET 0x8C0 > +#define DSDT_OFFSET 0x900 > -#define FADT_OFFSET 0x200 > -#define HPET_OFFSET 0x340 > -#define MCFG_OFFSET 0x380 > -#define FACS_OFFSET 0x3C0 > -#define DSDT_OFFSET 0x400 >=20 > I'll create a bug for this so that the overwrite will be detected at > run-time, and also bump up the space to allow for some growth. >=20 > later, >=20 > Peter. >=20 > * see the ACPI spec at http://www.acpi.info/spec.htm for table details.= > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" Ha, so the only reason I didn't run into this, is the biggest VM i've created just happened to be 20 vCPUs on my 24 CPU system. Dodged a bullet, or, delay the discovery of this bug by 2 years. One of the two. --=20 Allan Jude --VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC-- --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJX3eBEAAoJEBmVNT4SmAt+gqQP/iwcO7EJc8+8CV5RGXwETN4W xfXo1IvOJzKhmJwtUX+3d7UaWH8eLLpK8d0NIhenlcyKCivhJSQcjnTNkvcUVWiB roFCq3cKEZe116h3Y8on/rbjgmpdIjVP5mOOLZ4qulVVY/iogHVZpX0+WdORpt/q 6yGGurYbyoxQ4fZJcIGDI9A1Rjt7ON9bNkfjPvV8iJ66ZNFkOqUFmjifEq6pKW1w thOMBVWcV033BeRpYAQ41NjM56qF6ek2GXi56p7lxeZSif5yUXsCoEs7GhaZnKPs j19gJ0ofaYhsgNzYeiyMDcqDWufCJkUiiMwjpJVwQlG/5x/qWJpnZeHwOthn1mLE FGLHxOgQw3RU3QNop9N0XMEnsdWrmU2Su5ZqeLK2laT/39VTyf7OndSUavtUrPml zrDcnId14WoOSYMFGTZ9noJTUxtmDMTZBMQ0vNRPgNmm0/JTdOylTGnfY/WO0RaI 5oN9G7y9aIIu+DIvXsYI6Ri1IdqY4j2WSj+hbddiTiTmnBGS3/i0c/PkQ15wcRHj eXZYHRUHHdSbjfrXED423r/2pm/FP45OSit+YYBg17wNLqZ6aWWrW5aIKTHqcRV/ ZfksuMIvC3VOVai3McXSPi59pExp13iiaXg6iVnyt6QmxD6nLk0e5yvOPbsJt6UP VMucH+c0VWETOq60oxEy =eMLS -----END PGP SIGNATURE----- --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4ede10b9-95b2-f5ba-522e-de08a3937068>