Date: Fri, 29 Mar 2013 14:19:19 -0500 (CDT) From: Dan Mack <mack@macktronics.com> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve tty / login problems / panic Message-ID: <20130329140714.L12104@coco.macktronics.com> In-Reply-To: <515514E2.1080905@freebsd.org> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 28 Mar 2013, Peter Grehan wrote:
> Hi Dan,
>
>> Any ideas?
>
> Just to confirm: your host system is an i7 3930k, which looks like a 6 core
> system. Should be plenty beefy enough for a 2 vCPU guest.
That's what I thought, 6 cores, 12 threads, 64 GB RAM.
> The issue you are seeing looks a bit like what happens when systems are
> heavily oversubscribed and vCPUs can't get enough run time.
I was running in on an idle system; only bhyve was running.
> How much spare RAM is on the host when you started the vm ? Also, is there
> much running on the host when the guest is started ?
64GB
Here's some more information (sorry for the formatting):
Case 1: single vCPU via:
root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
Top shows it idle (note low load, 4GB memory given to guest, still 53GB free RAM, nothing else clocking CPU cycles):
root@olive:~ # top -bH
last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 14:11:51
24 processes: 1 running, 23 sleeping
Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free
ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other
Swap:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% bhyve{coco vcpu 0}
925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd
2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% bhyve{coco mevent}
So, case #1 all is normal. I was able to build a kernel with very good performance.
The boot sequence looks like the following:
root@olive:/vms/coco # ./vmrun.sh coco
Launching virtual machine "coco" ...
Consoles: userboot
FreeBSD/amd64 User boot, Revision 1.1
(root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
Loading /boot/defaults/loader.conf
/boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08]
/
______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s` `.....---.......--.``` -/
+------------Welcome to FreeBSD-----------+ +o .--` /y:` +.
| | yo`:. :o `+-
| 1. Boot Multi User [Enter] | y/ -/` -o/
| 2. Boot [S]ingle User | .- ::/sy+:.
| 3. [Esc]ape to loader prompt | / `-- /
| 4. Reboot | `: :`
| | `: :`
| Options: | / /
| 5. Configure Boot [O]ptions... | .- -.
| | -- -.
| | `:` `:`
| | .-- `--.
| | .---.....----.
+-----------------------------------------+
Booting...
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7
Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE>
Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 5368709120 (5120 MB)
avail memory = 3957534720 (3774 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BHYVE BVMADT >
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 1
ioapic0 <Version 1.1> irqs 0-16 on motherboard
kbd0 at kbdmux0
acpi0: <BHYVE BVXSDT> on motherboard
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: no PRT entry for 0.31.INTA
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
vtnet0: Ethernet address: 00:a0:98:ac:0d:df
virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
vtblk0: 8192MB (16777216 512 byte sectors)
uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0
uart2: console (9600,n,8,1)
Timecounters tick every 10.000 msec
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Timecounter "TSC-low" frequency 1599917752 Hz quality 1000
Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf.
Setting hostid: 0x24bfd025.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% fragmentation)
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Mounting local file systems:.
Writing entropy file:.
Setting hostname: cocopuff.example.com.
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:ac:0d:df
inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
Starting devd.
add net default: gateway 172.22.22.252
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Updating motd:.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Fri Mar 29 14:09:20 CDT 2013
FreeBSD/amd64 (cocopuff.example.com) (console)
login:
====== Next, boot the same image with two (2) vCPU:
Command run by vmrun.sh:
root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
Top shows it running out of control:
root@olive:~ # top -bH
last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 14:17:32
25 processes: 3 running, 22 sleeping
Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free
ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other
Swap:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% bhyve{coco vcpu 1}
64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% bhyve{coco vcpu 0}
15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd
807 root 20 0 13112K 4812K select 5 0:00 0.00% devd
925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd
1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron
2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
64163 root 52 0 4120M 120M kqread 5 0:00 0.00% bhyve{coco mevent}
15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh
Even though, it's just sitting at the login prompt:
root@olive:/vms/coco # ./vmrun.sh coco
Launching virtual machine "coco" ...
Consoles: userboot
FreeBSD/amd64 User boot, Revision 1.1
(root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
Loading /boot/defaults/loader.conf
/boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08]
/
______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s` `.....---.......--.``` -/
+------------Welcome to FreeBSD-----------+ +o .--` /y:` +.
| | yo`:. :o `+-
| 1. Boot Multi User [Enter] | y/ -/` -o/
| 2. Boot [S]ingle User | .- ::/sy+:.
| 3. [Esc]ape to loader prompt | / `-- /
| 4. Reboot | `: :`
| | `: :`
| Options: | / /
| 5. Configure Boot [O]ptions... | .- -.
| | -- -.
| | `:` `:`
| | .-- `--.
| | .---.....----.
+-----------------------------------------+
Booting...
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7
Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE>
Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 5368709120 (5120 MB)
avail memory = 3957534720 (3774 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BHYVE BVMADT >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-16 on motherboard
kbd0 at kbdmux0
acpi0: <BHYVE BVXSDT> on motherboard
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: no PRT entry for 0.31.INTA
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
vtnet0: Ethernet address: 00:a0:98:ac:0d:df
virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
vtblk0: 8192MB (16777216 512 byte sectors)
uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0
uart2: console (9600,n,8,1)
Timecounters tick every 10.000 msec
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf.
Setting hostid: 0x24bfd025.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% fragmentation)
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Mounting local file systems:.
Writing entropy file:.
Setting hostname: cocopuff.example.com.
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:ac:0d:df
inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
Starting devd.
add net default: gateway 172.22.22.252
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
syslogd: timed out waiting for child
/etc/rc: WARNING: failed to start syslogd
Clearing /tmp (X related).
Updating motd:.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Thu Apr 18 02:06:43 CDT 2013
FreeBSD/amd64 (cocopuff.example.com) (console)
login:
========== end of boot sequence
So, something is broken with SMP on the guest, I'd be happy to help just let me know.
Dan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130329140714.L12104>