Date: Fri, 29 Mar 2013 14:19:19 -0500 (CDT) From: Dan Mack <mack@macktronics.com> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve tty / login problems / panic Message-ID: <20130329140714.L12104@coco.macktronics.com> In-Reply-To: <515514E2.1080905@freebsd.org> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 28 Mar 2013, Peter Grehan wrote: > Hi Dan, > >> Any ideas? > > Just to confirm: your host system is an i7 3930k, which looks like a 6 core > system. Should be plenty beefy enough for a 2 vCPU guest. That's what I thought, 6 cores, 12 threads, 64 GB RAM. > The issue you are seeing looks a bit like what happens when systems are > heavily oversubscribed and vCPUs can't get enough run time. I was running in on an idle system; only bhyve was running. > How much spare RAM is on the host when you started the vm ? Also, is there > much running on the host when the guest is started ? 64GB Here's some more information (sorry for the formatting): Case 1: single vCPU via: root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it idle (note low load, 4GB memory given to guest, still 53GB free RAM, nothing else clocking CPU cycles): root@olive:~ # top -bH last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 14:11:51 24 processes: 1 running, 23 sleeping Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% bhyve{coco vcpu 0} 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% bhyve{coco mevent} So, case #1 all is normal. I was able to build a kernel with very good performance. The boot sequence looks like the following: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE> Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <BHYVE BVMADT > random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 1 ioapic0 <Version 1.1> irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: <BHYVE BVXSDT> on motherboard atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: <VirtIO Networking Adapter> on virtio_pci0 virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress> vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: <VirtIO Block Adapter> on virtio_pci1 virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs> vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet 1000baseT <full-duplex> status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Fri Mar 29 14:09:20 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ====== Next, boot the same image with two (2) vCPU: Command run by vmrun.sh: root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it running out of control: root@olive:~ # top -bH last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 14:17:32 25 processes: 3 running, 22 sleeping Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% bhyve{coco vcpu 1} 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% bhyve{coco vcpu 0} 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% bhyve{coco mevent} 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh Even though, it's just sitting at the login prompt: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE> Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <BHYVE BVMADT > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 2 ioapic0 <Version 1.1> irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: <BHYVE BVXSDT> on motherboard atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: <VirtIO Networking Adapter> on virtio_pci0 virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress> vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: <VirtIO Block Adapter> on virtio_pci1 virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs> vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet 1000baseT <full-duplex> status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: timed out waiting for child /etc/rc: WARNING: failed to start syslogd Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu Apr 18 02:06:43 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ========== end of boot sequence So, something is broken with SMP on the guest, I'd be happy to help just let me know. Dan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130329140714.L12104>