Date: Tue, 06 Mar 2018 07:45:00 +0100 From: "Fabian Freyer" <fabian.freyer@physik.tu-berlin.de> To: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org Subject: rumpkernel and bhyve: triple faults Message-ID: <C49D0E56-10A4-49D8-A843-E371395831B5@physik.tu-berlin.de>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 3156 and 4880).
--=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hello lists,
I=E2=80=99m currently playing around with getting rump kernels built with=
rumpkernel(7) running on FreeBSD=E2=80=99s bhyve(4). I=E2=80=99m using a=
custom boot loader [1] which builds on some patches to bhyveload / user =
boot [2].
To test, I=E2=80=99m using a simple =E2=80=9Chelloer=E2=80=9D unikernel f=
rom the tutorial [3]:
#include <stdio.h>
#include <unistd.h>
int
main()
{
printf("Hello, Rumprun ... I'm feeling tired\n");
sleep(2);
printf("much better!\n");
return 0;
}
I=E2=80=99m compiling this for the hw_virtio platform:
x86_64-rumprun-netbsd-gcc -o helloer-rumprun helloer.c
rumprun-bake hw_virtio helloer-rumprun.elf helloer-rumprun
Starting this kernel in bhyve gives me the following VM Exit:
=46rom the Intel SDM 3D, Appendix C [4], VM Exit 2 is a triple fault.
To further debug this, I=E2=80=99m using the ktr(4) support introduced in=
FreeBSD r276098 [5].
I=E2=80=99ve appended a full KTR trace further down in my mail, but I=E2=80=
=99ll go along the trace here with some commentary in execution order, an=
d attempt to annotate the source code lines from rumpkernel:
0x104000 is the entry address given in rump kernel=E2=80=99s multi boot h=
eader.
0000000000104000 <_start>:
632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000
636 0 350886961656 vm testing[0]: Resume execution at 0x104000
This is the cpuid instruction in x86_cpuid which is called from hyperviso=
r_detect:
rumprun/platform/hw/arch/x86/x86_subr.c: 91
0000000000102f90 <x86_cpuid>:
102f90: 49 89 d2 mov r10,rdx
102f93: 49 89 c9 mov r9,rcx
102f96: 53 push rbx
102f97: 89 f8 mov eax,edi
102f99: 0f a2 cpuid
rumprun/platform/hw/arch/x86/hypervisor.c: 38
637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8
638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f=
99
639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b
rumprun/platform/hw/arch/x86/hypervisor.c: 40
640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8
641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f=
99
642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b
rumprun/platform/hw/arch/x86/hypervisor.c: 51
643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8
644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f=
99
645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b
The next vexit is in cons_init, directly after the call to hypervisor_det=
ect.
0000000000102a50 <cons_init>:
102a50: 53 push rbx
102a51: e8 ea 0a 00 00 call 103540 <hypervisor_detect>=
102a56: 66 83 3d a4 d5 ef ff cmp WORD PTR [rip-0x102a5c],0x=
0 # 2 <current_lwp+0x2>
Due to compiler optimisations, the check here isn=E2=80=99t the
(hypervisor =3D=3D HYPERVISOR_XEN) check directly after the call to hyper=
visor_detect,
but the check (bios_crtc_base =3D=3D 0) in
rumprun/platform/hw/arch/x86/cons.c:59:
649 0 350887182668 vm testing[0]: handled exception vmexit at 0x=
102a56
Therefore, I=E2=80=99m assuming this is the origin of the fault.
Tracking down bios_crtc_base, I find that it=E2=80=99s loaded in
rumprun/platform/hw/arch/amd64/locore.S:70:
/* save BIOS data area values */
movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base
Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the b=
hyve
device node in /dev/vmm with xxd(1), I find the words at these addresses =
to be
Uninitialised:
00000400: 0000 ..
00000483: 0000 ..
I=E2=80=99m not sure where to go from here. Is this a bug in bhyve(4), sh=
ould these
values be initialised somehow, or should I patch rumpkernel(7) to skip th=
is check
when running on bhyve(4)?
Fabian
Full KTR trace:
index cpu timestamp trace
------ --- ---------------- -----
673 0 350887364560 vm testing[0]: vcpu state changed from frozen=
to idle
672 0 350887363766 vm testing[0]: retu 0/14
671 0 350887362932 vm testing[0]: All vcpus suspended
670 0 350887357212 vm testing[0]: vcpu state changed from runnin=
g to frozen
669 0 350887350444 vm testing[0]: returning from vmx_run: exitco=
de 14
668 0 350887348806 vm testing: virtual machine successfully susp=
ended 4
667 0 350887347712 vm testing[0]: triple fault: info1(0x80000b08=
), info2(0x80000b0e)
666 0 350887347590 vm testing[0]: Exception 14 delivered: 0x8000=
0b0e
665 0 350887346312 vm testing[0]: handled exception vmexit at 0x=
102a56
664 0 350887346178 vm testing[0]: Exception 14 pending
663 0 350887346060 vm testing[0]: Setting intr_shadow to 0 succe=
eded
662 0 350887341876 vm testing[0]: Reflecting exception 14/0 into=
the guest
661 0 350887339570 vm testing[0]: vm_exit_intinfo: info1(0x80000=
b08)
660 0 350887315326 vm testing[0]: Resume execution at 0x102a56
659 0 350887311168 vm testing[0]: vm_entry_intinfo: info1(0x8000=
0b0e), info2(0x80000b0e), retinfo(0x80000b08)
658 0 350887310988 vm testing[0]: Exception 14 delivered: 0x8000=
0b0e
657 0 350887309700 vm testing[0]: handled exception vmexit at 0x=
102a56
656 0 350887309570 vm testing[0]: Exception 14 pending
655 0 350887309442 vm testing[0]: Setting intr_shadow to 0 succe=
eded
654 0 350887305126 vm testing[0]: Reflecting exception 14/0 into=
the guest
653 0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x80000=
b0e)
652 0 350887248280 vm testing[0]: Resume execution at 0x102a56
651 0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), in=
fo2(0x80000b0e), retinfo(0x80000b0e)
650 0 350887184040 vm testing[0]: Exception 14 delivered: 0x8000=
0b0e
649 0 350887182668 vm testing[0]: handled exception vmexit at 0x=
102a56
648 0 350887182374 vm testing[0]: Exception 14 pending
647 0 350887182240 vm testing[0]: Setting intr_shadow to 0 succe=
eded
646 0 350887176682 vm testing[0]: Reflecting exception 14/0 into=
the guest
645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b
644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f=
99
643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8
642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b
641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f=
99
640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8
639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b
638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f=
99
637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8
636 0 350886961656 vm testing[0]: Resume execution at 0x104000
635 0 350886900428 vm testing[0]: vcpu state changed from frozen=
to running
634 0 350886895942 vm testing[0]: vcpu state changed from idle t=
o frozen
633 0 350886886630 vm testing[0]: vcpu state changed from frozen=
to idle
632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000
631 0 350886802596 vm testing[0]: vcpu state changed from idle t=
o frozen
630 3 350886414966 vm testing[0]: vcpu state changed from frozen=
to idle
629 3 350886414184 vm testing[0]: activated
628 3 350886412730 vm testing[0]: vcpu state changed from idle t=
o frozen
627 3 350886137831 vm testing[0]: vcpu state changed from frozen=
to idle
626 3 350885989045 vm testing[0]: vcpu state changed from idle t=
o frozen
625 3 350885773367 vm testing: RTC time set to 0x5a9e154e
624 3 350885773017 vm testing: Updating RTC base uptime from 0x3=
1c2762314 to 0x8bd6b051a2
623 3 350885772543 vm testing: Updating RTC secs from 0x5a9e14f4=
to 0x5a9e154e
622 3 350885017699 vm testing: RTC nvram write 0 to offset 0x5d
621 3 350885015953 vm testing: RTC nvram write 0 to offset 0x5c
620 3 350885013951 vm testing: RTC nvram write 0 to offset 0x5b
619 3 350885010615 vm testing: RTC nvram write 0x1 to offset 0x3=
5
618 3 350885008163 vm testing: RTC nvram write 0 to offset 0x34
[1] https://github.com/fubarnetes/bhyve-multiboot
[2] https://reviews.FreeBSD.org/D14473
[3] https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Building-Rumprun-=
Unikernels#2-building-applications
[4] https://www.intel.com/content/dam/www/public/us/en/documents/manuals/=
64-ia-32-architectures-software-developer-vol-3d-part-4-manual.pdf
[5] https://svnweb.freebsd.org/base?view=3Drevision&revision=3D276098
--=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_=
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename=signature.asc
Content-Type: application/pgp-signature; name=signature.asc
-----BEGIN PGP SIGNATURE-----
iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqeOOwiHGZhYmlhbi5m
cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5owLD/915dsZZNlz
stLP2tVgPifGtY8yCD5U/WfpWznOxzDmeyBlJszTdGoKRdbTDoZHSHsEype/PjAw
fn6cQKHChmsW7GwZSMZ/ETgu9xYzPaS9tDQPpRNcRpku0Npvw4Kifv4QzSibwP7e
6vfTFk59SoCkScinVXuKbjvylwnEtxbzIxWvSo24aXebcEdqSKsX8zKemGra00XM
S9NSny4Sa5VOpuMsPMhuvwexEP3CJW888h5/3xPNU9N5/vbLc1FDHarS2QNQF8fB
iy1F0lyIEP4iAzwO6STLtJv+tMiZh//2V/o6tKxFQMtgDlc6bb/vw40OlG/Z8Es/
Z9Gbeg6YZVdrxLGVE5sEgiC4BHkmgbFGp1yqkA23OMF3nFqTiLulftUun90S66uT
Vy2vor/tnVH6FrhnZ+f8Xh2s1movStz6DpEUFlZfmOxE6Z5lftn5089KYRZeMs++
i4P5x9ajhIhGNeqkKIeeWgjZ7CNh+CcG/ZJqvWmcln/j8/lPSIktCYSHPbCOYdzb
1RigTNstbX9I+2fZU4caaMlxcDE712zdt4HA55j7UbTPjeMRcXah1H/szPBcHWQK
O923YtXpiYRQaRTIAzt9++BWtjU60P2YQ4KQnZ2hzs6DlUKBSr33FFU4xUsFUt6Q
FrBzA4skxpa29GEsgeL2sreOREmNY6aQwg==
=qScN
-----END PGP SIGNATURE-----
--=_MailMate_F308673C-73D9-40FB-8BCF-C62E1950BE85_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C49D0E56-10A4-49D8-A843-E371395831B5>