From owner-freebsd-virtualization@FreeBSD.ORG Sun Mar 31 04:03:49 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 59FB87A7 for ; Sun, 31 Mar 2013 04:03:49 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id 1DC1E8A6 for ; Sun, 31 Mar 2013 04:03:49 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id qd14so1562089ieb.28 for ; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=iPtmAf7DhdubezAzbFR89VItpHQjebW4/0zFlbCUpuI=; b=ezB9im/A1AnWumUQhXXlkpITzfxB8fFYdsz7ogdUstVhUPt1CUpKQAPqucp4hlZIkG ASvtg+m4gikd3JO4V7kTb40YSggor92WXR/Yc8zDiMg/2VwxJksGLU8GYIci7/h3Nvo8 LjwiDX5OoppBCB8Bxt1ODR2Wz3IfteraPMJ+frCDRZcTn98kyan8CbI4P6yyolmIWJI1 4Cv4/r4Fgr0hWLcKzCj5SPkgluaTtL38FS5hHVml9HxPYmVmrjmReix5uuK0vG64zzaL ybwzQqkQo2u9Ig2NTll3kLA9Y1nS9ZYaSFA4z91+ycP54ONmGwE8CHX5g51Ys5cbcx99 mLkw== MIME-Version: 1.0 X-Received: by 10.42.145.137 with SMTP id f9mr3529427icv.52.1364702628690; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) In-Reply-To: <20130330141141.B17063@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> <20130330141141.B17063@coco.macktronics.com> Date: Sat, 30 Mar 2013 21:03:48 -0700 Message-ID: Subject: Re: bhyve tty / login problems / panic From: Neel Natu To: Dan Mack Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2013 04:03:49 -0000 Hi Dan, On Sat, Mar 30, 2013 at 12:17 PM, Dan Mack wrote: > Here you go: > > - uni-processor output: > > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 1330 > number of times hlt was intercepted 676 > number of times %cr access was intercepted 1 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 428499 > vm exits due to interrupt window opening 1231 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 423985 > number of times cpuid was intercepted 22 > vm exits due to nested page fault 7523 > number of vm exits for unknown reason 0 > number of times astpending at exit 38 > number of vm exits handled in userspace 853290 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 6954314530 > number of ticks vcpu was idle 59572 > vcpu migration across host cpus 29 > > number of times hlt was ignored 0 > total number of vm exits 863271 > > And here is when running with vcpu==2: > > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 19234 > number of times hlt was intercepted 1965 > number of times %cr access was intercepted 1 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 3907709 > vm exits due to interrupt window opening 8105 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 457998 > number of times cpuid was intercepted 50 > vm exits due to nested page fault 940909 > number of vm exits for unknown reason 0 > number of times astpending at exit 102 > number of vm exits handled in userspace 4367825 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 51834598990 > number of ticks vcpu was idle 887 > vcpu migration across host cpus 25 > > number of times hlt was ignored 0 > total number of vm exits 5335975 > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 > vcpu1 > vm exits due to external interrupt 51249 > number of times hlt was intercepted 3676 > number of times %cr access was intercepted 2 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 4667527 > vm exits due to interrupt window opening 32453 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 57278 > number of times cpuid was intercepted 63 > vm exits due to nested page fault 266091 > number of vm exits for unknown reason 0 > number of times astpending at exit 164 > number of vm exits handled in userspace 4728481 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 53833466076 > number of ticks vcpu was idle 965 > vcpu migration across host cpus 16 > > number of times hlt was ignored 0 > total number of vm exits 5078343 > > The panic comes eventually like this: > > login: panic: deadlkres: possible deadlock detected for > 0xfffffe0090fb5490, blocked for 217798 ticks > > cpuid = 0 > Uptime: 255d6h29m56s > Automatic reboot in 15 seconds - press a key on the console to abort > > Thanks for capturing that. In the SMP case the aberrant source of VM exits is the nested page table faults - which means local apic accesses since these are being entirely handled in the kernel. They are an order of magnitude more than in the UP case - even after adjusting for the difference in the run times in the two cases. Also, the uptime message (255d6h29m56s) is indicates something funky happening with the time keeping. I have added one more stat to vmm.ko that counts the number of times a local apic timer interrupt is generated for each vcpu. Also, I would like to install your release.iso if you are able to upload it. I very much doubt, it but want to rule out those specific bits as the source of the problem. Thanks again for helping to debug this. best Neel > > Hope this helps, and FYI, here's a dmesg from the host: > > > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013 > root@olive.example.com:/usr/**obj/usr/src/sys/MACKGEN amd64 > > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU) > > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d > Stepping = 7 > > Features=0xbfebfbff APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** > MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> > > Features2=0x1fbee3bf VMX,EST,TM2,SSSE3,CX16,xTPR,**PDCM,PCID,DCA,SSE4.1,SSE4.2,** > x2APIC,POPCNT,TSCDLT,AESNI,**XSAVE,OSXSAVE,AVX> > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 68719476736 (65536 MB) > avail memory = 63123800064 (60199 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs > FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads > > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 6 > cpu7 (AP): APIC ID: 7 > cpu8 (AP): APIC ID: 8 > cpu9 (AP): APIC ID: 9 > cpu10 (AP): APIC ID: 10 > cpu11 (AP): APIC ID: 11 > ioapic1 irqs 24-47 on motherboard > ioapic0 irqs 0-23 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: Power Button (fixed) > acpi0: reservation of f0000000, 18000000 (3) failed > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 > > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > hpet0: iomem 0xfed00000-0xfed03fff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 340 > Event timer "HPET2" frequency 14318180 Hz quality 340 > Event timer "HPET3" frequency 14318180 Hz quality 340 > Event timer "HPET4" frequency 14318180 Hz quality 340 > Event timer "HPET5" frequency 14318180 Hz quality 340 > Event timer "HPET6" frequency 14318180 Hz quality 340 > Event timer "HPET7" frequency 14318180 Hz quality 340 > > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > acpi_button0: on acpi0 > > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 1.0 on pci0 > pci1: on pcib1 > pcib2: at device 1.1 on pci0 > pci2: on pcib2 > xhci0: mem 0xe3400000-0xe3401fff irq 16 > at device 0.0 on pci2 > usbus0: waiting for BIOS to give up control > xhci0: 32 byte context size. > usbus0 on xhci0 > pcib3: at device 2.0 on pci0 > pci3: on pcib3 > vgapci0: port 0x3000-0x307f mem > 0xe2000000-0xe2ffffff,**0xd0000000-0xdfffffff,**0xe0000000-0xe1ffffff irq > 16 at device 0.0 on pci3 > pci3: at device 0.1 (no driver attached) > pcib4: at device 3.0 on pci0 > pci4: on pcib4 > pci0: at device 5.0 (no driver attached) > pci0: at device 5.2 (no driver attached) > pcib5: at device 17.0 on pci0 > pci5: on pcib5 > pci0: at device 22.0 (no driver attached) > em0: port 0x4040-0x405f mem > 0xe3500000-0xe351ffff,**0xe3521000-0xe3521fff irq 20 at device 25.0 on > pci0 > em0: Using an MSI interrupt > em0: Ethernet address: 4c:72:b9:d1:b2:d2 > ehci0: mem 0xe3601000-0xe36013ff irq > 16 at device 26.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci0 > pcib6: at device 28.0 on pci0 > pci6: on pcib6 > pcib7: at device 28.2 on pci0 > pci7: on pcib7 > pci7: at device 0.0 (no driver attached) > pcib8: at device 28.3 on pci0 > pci8: on pcib8 > xhci1: mem 0xe3200000-0xe3201fff irq 19 > at device 0.0 on pci8 > usbus2: waiting for BIOS to give up control > xhci1: 32 byte context size. > usbus2 on xhci1 > pcib9: at device 28.5 on pci0 > pci9: on pcib9 > em1: port 0x1000-0x101f mem > 0xe3100000-0xe311ffff,**0xe3120000-0xe3123fff irq 17 at device 0.0 on pci9 > em1: Using MSIX interrupts with 3 vectors > em1: Ethernet address: 4c:72:b9:d1:b2:d3 > ehci1: mem 0xe3602000-0xe36023ff irq > 23 at device 29.0 on pci0 > usbus3: EHCI version 1.0 > usbus3 on ehci1 > pcib10: at device 30.0 on pci0 > pci10: on pcib10 > isab0: at device 31.0 on pci0 > isa0: on isab0 > ahci0: port > 0x4068-0x406f,0x4074-0x4077,**0x4060-0x4067,0x4070-0x4073,**0x4020-0x403f > mem 0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0 > ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahcich2: at channel 2 on ahci0 > ahcich3: at channel 3 on ahci0 > ahcich4: at channel 4 on ahci0 > ahcich5: at channel 5 on ahci0 > ahciem0: on ahci0 > pci0: at device 31.3 (no driver attached) > orm0: at iomem 0xd1000-0xd1fff,0xd2000-**0xd2fff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > ppc0: cannot reserve I/O port range > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > est2: on cpu2 > p4tcc2: on cpu2 > est3: on cpu3 > p4tcc3: on cpu3 > est4: on cpu4 > p4tcc4: on cpu4 > est5: on cpu5 > p4tcc5: on cpu5 > est6: on cpu6 > p4tcc6: on cpu6 > est7: on cpu7 > p4tcc7: on cpu7 > est8: on cpu8 > p4tcc8: on cpu8 > est9: on cpu9 > p4tcc9: on cpu9 > est10: on cpu10 > p4tcc10: on cpu10 > est11: on cpu11 > p4tcc11: on cpu11 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > usbus0: 5.0Gbps Super Speed USB v3.0 > usbus1: 480Mbps High Speed USB v2.0 > usbus2: 5.0Gbps Super Speed USB v3.0 > usbus3: 480Mbps High Speed USB v2.0 > ugen1.1: at usbus1 > uhub0: on usbus1 > ugen0.1: <0x1033> at usbus0 > uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 > ugen3.1: at usbus3 > uhub2: on usbus3 > ugen2.1: <0x1033> at usbus2 > uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 > ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 > ses0: SEMB S-E-S 2.00 device > ses0: SEMB SES Device > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ATA-9 SATA 3.x device > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad4 > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > ada1: ATA-9 SATA 3.x device > ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada1: Command Queueing enabled > ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) > ada1: Previously was known as ad6 > ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 > ada2: ATA-8 SATA 3.x device > ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada2: Command Queueing enabled > ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada2: Previously was known as ad8 > ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 > ada3: ATA-8 SATA 3.x device > ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada3: Command Queueing enabled > ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada3: Previously was known as ad10 > ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 > ada4: ATA-8 SATA 3.x device > ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada4: Command Queueing enabled > ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada4: Previously was known as ad12 > ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 > ada5: ATA-8 SATA 3.x device > ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada5: Command Queueing enabled > ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada5: Previously was known as ad14 > lapic1: Forcing LINT1 to edge trigger > > SMP: AP CPU #1 Launched! > lapic4: Forcing LINT1 to edge trigger > SMP: AP CPU #4 Launched! > lapic11: Forcing LINT1 to edge trigger > SMP: AP CPU #11 Launched! > lapic7: Forcing LINT1 to edge trigger > SMP: AP CPU #7 Launched! > lapic5: Forcing LINT1 to edge trigger > SMP: AP CPU #5 Launched! > lapic9: Forcing LINT1 to edge trigger > SMP: AP CPU #9 Launched! > lapic2: Forcing LINT1 to edge trigger > SMP: AP CPU #2 Launched! > lapic3: Forcing LINT1 to edge trigger > SMP: AP CPU #3 Launched! > lapic10: Forcing LINT1 to edge trigger > SMP: AP CPU #10 Launched! > lapic6: Forcing LINT1 to edge trigger > SMP: AP CPU #6 Launched! > lapic8: Forcing LINT1 to edge trigger > SMP: AP CPU #8 Launched! > uhub3: 4 ports with 4 removable, self powered > uhub1: 4 ports with 4 removable, self powered > Root mount waiting for: usbus3 usbus1 > uhub0: 2 ports with 2 removable, self powered > uhub2: 2 ports with 2 removable, self powered > Root mount waiting for: usbus3 usbus1 > ugen1.2: at usbus1 > uhub4: on > usbus1 > ugen3.2: at usbus3 > uhub5: on > usbus3 > Root mount waiting for: usbus3 usbus1 > uhub4: 6 ports with 6 removable, self powered > uhub5: 8 ports with 8 removable, self powered > ugen1.3: at usbus1 > ukbd0: on > usbus1 > kbd2 at ukbd0 > ugen3.3: at usbus3 > Root mount waiting for: usbus3 > ugen3.4: at usbus3 > Trying to mount root from zfs:tron []... > ums0: on > usbus1 > ums0: 5 buttons and [XYZ] coordinates ID=0 > run0: <1.0> on usbus3 > run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address > 94:db:c9:e3:6e:c8 > ubt0: > on usbus3 > WARNING: attempt to domain_add(bluetooth) after domainfinalize() > WARNING: attempt to domain_add(netgraph) after domainfinalize() > tap0: Ethernet address: 00:bd:8c:54:07:00 > tap0: link state changed to UP > bridge0: Ethernet address: 02:4c:89:ce:33:00 > tap0: promiscuous mode enabled > bridge0: link state changed to UP > em1: link state changed to DOWN > em1: promiscuous mode enabled > em1: link state changed to UP > tap0: link state changed to DOWN > tap0: link state changed to UP > tap0: link state changed to DOWN > tap0: link state changed to UP > > > > >> Yep, no problem. I saw the checkin and started a rebuild a few minutes >> ago :-) >> >> Thanks, >> >> Dan >> >> On Sat, 30 Mar 2013, Neel Natu wrote: >> >> Hi Dan, >>> >>> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: >>> >>> >>>> >>>> Here you go Neel: >>>> >>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >>>> vcpu0 >>>> vm exits due to external interrupt 45051 >>>> number of times hlt was intercepted 2277 >>>> number of NMIs delivered to vcpu 0 >>>> vcpu total runtime 105001652482 >>>> number of ticks vcpu was idle 919 >>>> vcpu migration across host cpus 13 >>>> number of times hlt was ignored 0 >>>> total number of vm exits 9240925 >>>> root@olive:~ # >>>> root@olive:~ # >>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >>>> vcpu1 >>>> vm exits due to external interrupt 149431 >>>> number of times hlt was intercepted 4222 >>>> number of NMIs delivered to vcpu 0 >>>> vcpu total runtime 106876622528 >>>> number of ticks vcpu was idle 869 >>>> vcpu migration across host cpus 20 >>>> number of times hlt was ignored 0 >>>> total number of vm exits 8065909 >>>> >>>> >>>> That did not help a whole lot because we were not keeping track of all >>> the >>> reasons a vcpu could exit. >>> >>> I have fixed that in r248935: >>> http://svnweb.freebsd.org/**base?view=revision&revision=**248935 >>> >>> Do you mind updating your vmm.ko with the change and getting the stats >>> again? >>> >>> best >>> Neel >>> >>> >>> Dan >>>> >>>> >>>> >>>> On Fri, 29 Mar 2013, Neel Natu wrote: >>>> >>>> Hi Dan, >>>> >>>>> >>>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack >>>>> wrote: >>>>> >>>>> >>>>> I ran the procsystime dtrace script on bhyve when it was chewing up >>>>>> all >>>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of >>>>>> runtime: >>>>>> >>>>>> root@olive:/usr/share/dtrace/******toolkit # ./procsystime -n bhyve >>>>>> -aT >>>>>> >>>>>> Tracing... Hit Ctrl-C to end... >>>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>>>> ^C >>>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>>>> >>>>>> Elapsed Times for processes bhyve, >>>>>> >>>>>> SYSCALL TIME (ns) >>>>>> _umtx_op 18349 >>>>>> writev 135712 >>>>>> preadv 16175267 >>>>>> pwritev 22924378 >>>>>> ioctl 4353897920 >>>>>> TOTAL: 4393151626 >>>>>> >>>>>> CPU Times for processes bhyve, >>>>>> >>>>>> SYSCALL TIME (ns) >>>>>> _umtx_op 8815 >>>>>> writev 103145 >>>>>> pwritev 10647023 >>>>>> preadv 15159976 >>>>>> ioctl 3943399889 >>>>>> TOTAL: 3969318848 >>>>>> >>>>>> Syscall Counts for processes bhyve, >>>>>> >>>>>> SYSCALL COUNT >>>>>> _umtx_op 17 >>>>>> writev 43 >>>>>> pwritev 639 >>>>>> preadv 1183 >>>>>> ioctl 652782 >>>>>> TOTAL: 654664 >>>>>> >>>>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>>>> >>>>>> Dan >>>>>> >>>>>> root@olive:/usr/share/dtrace/******toolkit # ./hotkernel >>>>>> >>>>>> Sampling... Hit Ctrl-C to end. >>>>>> ^C >>>>>> FUNCTION COUNT PCNT >>>>>> zfs.ko`space_map_sync 1 0.0% >>>>>> kernel`syscall_thread_exit 1 0.0% >>>>>> kernel`should_yield 1 0.0% >>>>>> zfs.ko`lzjb_compress 1 0.0% >>>>>> zfs.ko`dbuf_rele 1 0.0% >>>>>> zfs.ko`dsl_pool_tempreserve_******space 1 >>>>>> 0.0% >>>>>> >>>>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>>>> kernel`cpu_idleclock 1 0.0% >>>>>> kernel`_rw_runlock_cookie 1 0.0% >>>>>> kernel`pmap_enter 1 0.0% >>>>>> kernel`__lockmgr_args 1 0.0% >>>>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>>>> kernel`ahci_ch_intr 1 0.0% >>>>>> kernel`__rw_rlock 1 0.0% >>>>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>>>> zfs.ko`arc_buf_thaw 1 0.0% >>>>>> kernel`atomic_set_int 1 0.0% >>>>>> kernel`g_io_schedule_down 1 0.0% >>>>>> kernel`__mtx_lock_flags 1 0.0% >>>>>> kernel`_callout_stop_safe 1 0.0% >>>>>> kernel`sched_idletd 1 0.0% >>>>>> kernel`_sx_slock 1 0.0% >>>>>> vmm.ko`lapic_set_intr 1 0.0% >>>>>> kernel`cpusetobj_ffs 1 0.0% >>>>>> zfs.ko`dbuf_hold 1 0.0% >>>>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>>>> zfs.ko`sa_attr_op 1 0.0% >>>>>> zfs.ko`dmu_buf_hold_array_by_******dnode 1 >>>>>> 0.0% >>>>>> >>>>>> zfs.ko`taskq_run_safe 1 0.0% >>>>>> kernel`vm_reserv_free_page 1 0.0% >>>>>> kernel`_sx_sunlock 1 0.0% >>>>>> kernel`vn_rlimit_fsize 1 0.0% >>>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>>>> kernel`vn_write 1 0.0% >>>>>> kernel`lock_init 1 0.0% >>>>>> vmm.ko`vmx_resume 1 0.0% >>>>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>>>> kernel`mnt_vnode_next_active 1 0.0% >>>>>> kernel`vm_page_remove 1 0.0% >>>>>> zfs.ko`vdev_queue_deadline_******compare 1 >>>>>> 0.0% >>>>>> >>>>>> kernel`zone_free_item 1 0.0% >>>>>> vmm.ko`vmx_launch 1 0.0% >>>>>> kernel`m_pullup 1 0.0% >>>>>> kernel`pmap_change_wiring 1 0.0% >>>>>> kernel`sched_userret 1 0.0% >>>>>> zfs.ko`dbuf_write_done 1 0.0% >>>>>> zfs.ko`zio_execute 1 0.0% >>>>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>>>> zfs.ko`avl_rotation 2 0.0% >>>>>> kernel`ehci_device_isoc_fs_******enter 2 >>>>>> 0.0% >>>>>> >>>>>> vmm.ko`vm_exitinfo 2 0.0% >>>>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>>>> vmm.ko`vcpu_stats 2 0.0% >>>>>> kernel`atomic_add_long 2 0.0% >>>>>> vmm.ko`0xffffffff81 2 0.0% >>>>>> kernel`vm_reserv_alloc_page 2 0.0% >>>>>> kernel`pagezero 2 0.0% >>>>>> kernel`bzero 2 0.0% >>>>>> zfs.ko`fletcher_2_native 2 0.0% >>>>>> vmm.ko`vmx_getreg 2 0.0% >>>>>> vmm.ko`vm_guest_msrs 3 0.0% >>>>>> vmm.ko`vmx_setreg 3 0.0% >>>>>> vmm.ko`vm_get_register 3 0.0% >>>>>> kernel`sx_try_xlock_ 3 0.0% >>>>>> kernel`vm_radix_lookup_le 3 0.0% >>>>>> vmm.ko`lapic_pending_intr 4 0.0% >>>>>> kernel`syscall_thread_enter 4 0.0% >>>>>> kernel`cpu_set_syscall_retval 4 0.0% >>>>>> kernel`__mnt_vnode_next_all 4 0.0% >>>>>> kernel`vfs_stdsync 4 0.0% >>>>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>>>> kernel`_sx_xunlock 7 0.0% >>>>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>>>> vmm.ko`lapic_timer_tick 8 0.0% >>>>>> zfs.ko`list_next 8 0.0% >>>>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>>>> kernel`userret 9 0.0% >>>>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>>>> vmm.ko`vm_lapic 11 0.0% >>>>>> kernel`spinlock_enter 14 0.0% >>>>>> kernel`__mtx_lock_sleep 15 0.0% >>>>>> vmm.ko`vcpu_set_state 16 0.0% >>>>>> vmm.ko`vm_nmi_pending 16 0.0% >>>>>> kernel`malloc_type_zone_******allocated 16 >>>>>> 0.0% >>>>>> >>>>>> kernel`devfs_ioctl_f 16 0.0% >>>>>> kernel`pmap_kextract 17 0.0% >>>>>> kernel`sys_ioctl 18 0.0% >>>>>> kernel`malloc 21 0.0% >>>>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>>>> kernel`uma_zfree_arg 28 0.0% >>>>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>>>> kernel`trap 30 0.0% >>>>>> kernel`hpet_get_timecount 31 0.0% >>>>>> dtrace.ko`dtrace_trap 35 0.0% >>>>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>>>> kernel`copyout 43 0.0% >>>>>> kernel`critical_enter 44 0.0% >>>>>> kernel`copyin 51 0.1% >>>>>> vmm.ko`vmcs_read 53 0.1% >>>>>> kernel`fpusave 61 0.1% >>>>>> kernel`uma_zalloc_arg 62 0.1% >>>>>> kernel`devvn_refthread 69 0.1% >>>>>> kernel`atomic_load_acq_int 70 0.1% >>>>>> kernel`fpurestore 70 0.1% >>>>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>>>> kernel`amd64_syscall 77 0.1% >>>>>> kernel`bcopy 91 0.1% >>>>>> kernel`fpudna 103 0.1% >>>>>> kernel`free 103 0.1% >>>>>> kernel`critical_exit 105 0.1% >>>>>> kernel`dev_relthread 108 0.1% >>>>>> kernel`spinlock_exit 119 0.1% >>>>>> kernel`kern_ioctl 138 0.1% >>>>>> kernel`fpuexit 242 0.3% >>>>>> kernel`0xffffffff80 306 0.3% >>>>>> vmm.ko`restore_host_msrs 398 0.4% >>>>>> vmm.ko`vm_run 428 0.4% >>>>>> vmm.ko`restore_guest_msrs 433 0.5% >>>>>> kernel`cpu_idle 906 1.0% >>>>>> vmm.ko`vmx_run 9266 9.7% >>>>>> kernel`acpi_cpu_c1 81134 85.2% >>>>>> >>>>>> Hope this helps, let me know if you have any ideas on what to try >>>>>> next. >>>>>> >>>>>> >>>>>> Could you try the following command to get vcpu stats? >>>>>> >>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>>>> >>>>> best >>>>> Neel >>>>> >>>>> >>>>> Dan >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>>>> >>>>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>>>> >>>>>> >>>>>>> Hi Dan, >>>>>>> >>>>>>> >>>>>>>> Any ideas? >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Just to confirm: your host system is an i7 3930k, which looks >>>>>>>> like a 6 >>>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>>>> >>>>>>>> >>>>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>>>> >>>>>>> The issue you are seeing looks a bit like what happens when systems >>>>>>> are >>>>>>> >>>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>>>> >>>>>>>> >>>>>>>> I was running in on an idle system; only bhyve was running. >>>>>>> >>>>>>> >>>>>>> How much spare RAM is on the host when you started the vm ? Also, is >>>>>>> >>>>>>> there much running on the host when the guest is started ? >>>>>>>> >>>>>>>> >>>>>>>> 64GB >>>>>>> >>>>>>> Here's some more information (sorry for the formatting): >>>>>>> >>>>>>> Case 1: single vCPU via: >>>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>>> >>>>>>> Top shows it idle (note low load, 4GB memory given to guest, still >>>>>>> 53GB >>>>>>> free RAM, nothing else clocking CPU cycles): >>>>>>> >>>>>>> root@olive:~ # top -bH >>>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>>>> 14:11:51 >>>>>>> 24 processes: 1 running, 23 sleeping >>>>>>> >>>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>>>>> Swap: >>>>>>> >>>>>>> >>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>> COMMAND >>>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>>>> bhyve{coco vcpu 0} >>>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% >>>>>>> syslogd >>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>>>> bhyve{coco mevent} >>>>>>> >>>>>>> >>>>>>> So, case #1 all is normal. I was able to build a kernel with very >>>>>>> good >>>>>>> performance. >>>>>>> >>>>>>> The boot sequence looks like the following: >>>>>>> >>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>> Launching virtual machine "coco" ... >>>>>>> Consoles: userboot >>>>>>> >>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>> >>>>>>> / >>>>>>> ______ ____ _____ _____ >>>>>>> | ____| | _ \ / ____| __ \ >>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>> | | | | | | || | | | >>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>> ` >>>>>>> s` `.....---.......--.``` >>>>>>> -/ >>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>> +. >>>>>>> | | yo`:. :o >>>>>>> `+- >>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>> -o/ >>>>>>> | 2. Boot [S]ingle User | .- >>>>>>> ::/sy+:. >>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>> `-- >>>>>>> / >>>>>>> | 4. Reboot | `: >>>>>>> :` >>>>>>> | | `: >>>>>>> :` >>>>>>> | Options: | / >>>>>>> / >>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>> -. >>>>>>> | | -- -. >>>>>>> | | `:` >>>>>>> `:` >>>>>>> | | .-- `--. >>>>>>> | | .---.....----. >>>>>>> +-----------------------------******------------+ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Booting... >>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>> 1994 >>>>>>> The Regents of the University of California. All rights >>>>>>> reserved. >>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>> >>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class >>>>>>> CPU) >>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>> Stepping = 7 >>>>>>> Features=0x8fa3ab7f>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>> Features2=0x83bee217>>>>>> **** >>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>> TSCDLT,**AESNI,HV> >>>>>>> AMD Features=0x2c100800 >>>>>>> >>>>>>> AMD Features2=0x1 >>>>>>> TSC: P-state invariant >>>>>>> real memory = 5368709120 (5120 MB) >>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>> Event timer "LAPIC" quality 400 >>>>>>> ACPI APIC Table: >>>>>>> random device not loaded; using insecure entropy >>>>>>> ioapic0: Changing APIC ID to 1 >>>>>>> ioapic0 irqs 0-16 on motherboard >>>>>>> kbd0 at kbdmux0 >>>>>>> acpi0: on motherboard >>>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>>> pci0: on pcib0 >>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>> vtnet0: on virtio_pci0 >>>>>>> virtio_pci0: host features: 0x18020 >>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>> >>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>> vtblk0: on virtio_pci1 >>>>>>> virtio_pci1: host features: 0x10000004 >>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>> >>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>>> pci0 >>>>>>> uart2: console (9600,n,8,1) >>>>>>> Timecounters tick every 10.000 msec >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>> >>>>>>> Setting hostid: 0x24bfd025. >>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>> Starting file system checks: >>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>>>> fragmentation) >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Mounting local file systems:. >>>>>>> Writing entropy file:. >>>>>>> Setting hostname: cocopuff.example.com. >>>>>>> vtnet0: link state changed to UP >>>>>>> Starting Network: lo0 vtnet0. >>>>>>> lo0: flags=8049 metric 0 mtu >>>>>>> 16384 >>>>>>> options=600003 >>>>>>> >>>>>>> inet6 ::1 prefixlen 128 >>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>> nd6 options=21 >>>>>>> vtnet0: flags=8943>>>>>> MULTICAST> >>>>>>> metric 0 mtu 1500 >>>>>>> options=80028 >>>>>>> >>>>>>> ether 00:a0:98:ac:0d:df >>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>> 172.22.255.255 >>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>>> nd6 options=29 >>>>>>> >>>>>>> media: Ethernet 1000baseT >>>>>>> status: active >>>>>>> Starting devd. >>>>>>> add net default: gateway 172.22.22.252 >>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>> add net fe80::: gateway ::1 >>>>>>> add net ff02::: gateway ::1 >>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>> Creating and/or trimming log files. >>>>>>> Starting syslogd. >>>>>>> Clearing /tmp (X related). >>>>>>> Updating motd:. >>>>>>> Performing sanity check on sshd configuration. >>>>>>> Starting sshd. >>>>>>> Starting cron. >>>>>>> Starting background file system checks in 60 seconds. >>>>>>> >>>>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>>>> >>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>> >>>>>>> login: >>>>>>> >>>>>>> >>>>>>> ====== Next, boot the same image with two (2) vCPU: >>>>>>> >>>>>>> Command run by vmrun.sh: >>>>>>> >>>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>>> >>>>>>> Top shows it running out of control: >>>>>>> >>>>>>> root@olive:~ # top -bH >>>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>>>> 14:17:32 >>>>>>> 25 processes: 3 running, 22 sleeping >>>>>>> >>>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M >>>>>>> Other >>>>>>> Swap: >>>>>>> >>>>>>> >>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>> COMMAND >>>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>>>> bhyve{coco vcpu 1} >>>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>>>> bhyve{coco vcpu 0} >>>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% >>>>>>> syslogd >>>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>>>> bhyve{coco mevent} >>>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>>>> >>>>>>> Even though, it's just sitting at the login prompt: >>>>>>> >>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>> Launching virtual machine "coco" ... >>>>>>> Consoles: userboot >>>>>>> >>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>> >>>>>>> / >>>>>>> ______ ____ _____ _____ >>>>>>> | ____| | _ \ / ____| __ \ >>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>> | | | | | | || | | | >>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>> ` >>>>>>> s` `.....---.......--.``` >>>>>>> -/ >>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>> +. >>>>>>> | | yo`:. :o >>>>>>> `+- >>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>> -o/ >>>>>>> | 2. Boot [S]ingle User | .- >>>>>>> ::/sy+:. >>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>> `-- >>>>>>> / >>>>>>> | 4. Reboot | `: >>>>>>> :` >>>>>>> | | `: >>>>>>> :` >>>>>>> | Options: | / >>>>>>> / >>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>> -. >>>>>>> | | -- -. >>>>>>> | | `:` >>>>>>> `:` >>>>>>> | | .-- `--. >>>>>>> | | .---.....----. >>>>>>> +-----------------------------******------------+ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Booting... >>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>> 1994 >>>>>>> The Regents of the University of California. All rights >>>>>>> reserved. >>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>> >>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class >>>>>>> CPU) >>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>> Stepping = 7 >>>>>>> Features=0x8fa3ab7f>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>> Features2=0x83bee217>>>>>> **** >>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>> TSCDLT,**AESNI,HV> >>>>>>> AMD Features=0x2c100800 >>>>>>> >>>>>>> AMD Features2=0x1 >>>>>>> TSC: P-state invariant >>>>>>> real memory = 5368709120 (5120 MB) >>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>> Event timer "LAPIC" quality 400 >>>>>>> ACPI APIC Table: >>>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>>>> cpu0 (BSP): APIC ID: 0 >>>>>>> cpu1 (AP): APIC ID: 1 >>>>>>> random device not loaded; using insecure entropy >>>>>>> ioapic0: Changing APIC ID to 2 >>>>>>> ioapic0 irqs 0-16 on motherboard >>>>>>> kbd0 at kbdmux0 >>>>>>> acpi0: on motherboard >>>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>>> pci0: on pcib0 >>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>> vtnet0: on virtio_pci0 >>>>>>> virtio_pci0: host features: 0x18020 >>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>> >>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>> vtblk0: on virtio_pci1 >>>>>>> virtio_pci1: host features: 0x10000004 >>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>> >>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>>> pci0 >>>>>>> uart2: console (9600,n,8,1) >>>>>>> Timecounters tick every 10.000 msec >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> SMP: AP CPU #1 Launched! >>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>> >>>>>>> Setting hostid: 0x24bfd025. >>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>> Starting file system checks: >>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>>>> fragmentation) >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Mounting local file systems:. >>>>>>> Writing entropy file:. >>>>>>> Setting hostname: cocopuff.example.com. >>>>>>> vtnet0: link state changed to UP >>>>>>> Starting Network: lo0 vtnet0. >>>>>>> lo0: flags=8049 metric 0 mtu >>>>>>> 16384 >>>>>>> options=600003 >>>>>>> >>>>>>> inet6 ::1 prefixlen 128 >>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>> nd6 options=21 >>>>>>> vtnet0: flags=8943>>>>>> MULTICAST> >>>>>>> metric 0 mtu 1500 >>>>>>> options=80028 >>>>>>> >>>>>>> ether 00:a0:98:ac:0d:df >>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>> 172.22.255.255 >>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>>> nd6 options=29 >>>>>>> >>>>>>> media: Ethernet 1000baseT >>>>>>> status: active >>>>>>> Starting devd. >>>>>>> add net default: gateway 172.22.22.252 >>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>> add net fe80::: gateway ::1 >>>>>>> add net ff02::: gateway ::1 >>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>> Creating and/or trimming log files. >>>>>>> Starting syslogd. >>>>>>> syslogd: timed out waiting for child >>>>>>> /etc/rc: WARNING: failed to start syslogd >>>>>>> Clearing /tmp (X related). >>>>>>> Updating motd:. >>>>>>> Performing sanity check on sshd configuration. >>>>>>> Starting sshd. >>>>>>> Starting cron. >>>>>>> Starting background file system checks in 60 seconds. >>>>>>> >>>>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>>>> >>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>> >>>>>>> login: >>>>>>> >>>>>>> >>>>>>> ========== end of boot sequence >>>>>>> >>>>>>> So, something is broken with SMP on the guest, I'd be happy to help >>>>>>> just >>>>>>> let me know. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> ______________________________******_________________ >>>>>>> freebsd-virtualization@****freeb**sd.org < >>>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>>> freebsd-virtualization@**freebsd.org >>>>>>> >>mailing >>>>>>> list >>>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-**** >>>>>>> virtualization>>>>>> listinfo/freebsd-****virtualization >>>>>>> > >>>>>>> >>>>>> virtualization >>>>>>> >>>>>> freebsd-virtualization >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>>> unsubscribe@freebsd.org >>>>>> unsubscribe@freebsd.org>>>>>> freebsd.org > >>>>>>> >>>>>>>> " >>>>>>>> >>>>>>> >>>>>>> ______________________________******_________________ >>>>>>> >>>>>>> freebsd-virtualization@****freeb**sd.org < >>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>> freebsd-virtualization@**freebsd.org >>>>>> >>mailing >>>>>> list >>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-****** >>>>>> virtualization >>>>>> >>>>> virtualization >>>>>> > >>>>>> >>>>> virtualization >>>>>> >>>>>> > >>>>>> >>>>>>> >>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>>> > >>>>>> >>>>>>> " >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> ______________________________**_________________ >> freebsd-virtualization@**freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-** >> unsubscribe@freebsd.org " >> >> From owner-freebsd-virtualization@FreeBSD.ORG Sun Mar 31 15:47:31 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E2429E71 for ; Sun, 31 Mar 2013 15:47:31 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 9FB62155 for ; Sun, 31 Mar 2013 15:47:31 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id EDD134AC40; Sun, 31 Mar 2013 10:47:24 -0500 (CDT) Date: Sun, 31 Mar 2013 10:47:24 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: Message-ID: <20130331103902.V30526@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> <20130330141141.B17063@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2013 15:47:31 -0000 Here's the latest (note I didn't update my release.iso with the latest, at this point I'm still running a slightly earlier (cd /usr/src/release ; make release) copy from 248935 below. Also, the kernel that the VM is running I actually built while in uni-processor mode which shows how well behaved it is when running in non-SMP. You can see my motherboard type and such from the previous host boot log (Intel desktop board), let me know if you want any ACPI bios setting details; the firmware from Intel is about 1 month old so maybe it is related. Hope this helps, let me know where to upload my release.iso and/or a small disk image to boot if needed. Dan root@olive:/vms # bhyvectl --vm=coco --get-stats --cpu=0 ; bhyvectl --vm=coco --get-stats --cpu=1 vcpu0 vm exits due to external interrupt 29118 number of times hlt was intercepted 3410 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 7967499 vm exits due to interrupt window opening 11701 vm exits due to nmi window opening 0 number of times in/out was intercepted 480563 number of times cpuid was intercepted 122 vm exits due to nested page fault 1490965 number of vm exits for unknown reason 0 number of times astpending at exit 282 number of vm exits handled in userspace 8451625 number of NMIs delivered to vcpu 0 vcpu total runtime 94121692198 number of ticks vcpu was idle 5750 timer interrupts generated by vlapic 9073 vcpu migration across host cpus 52 number of times hlt was ignored 0 total number of vm exits 9983383 vcpu1 vm exits due to external interrupt 73725 number of times hlt was intercepted 4995 number of times %cr access was intercepted 2 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 7707742 vm exits due to interrupt window opening 45746 vm exits due to nmi window opening 0 number of times in/out was intercepted 79601 number of times cpuid was intercepted 105 vm exits due to nested page fault 384793 number of vm exits for unknown reason 0 number of times astpending at exit 263 number of vm exits handled in userspace 7792338 number of NMIs delivered to vcpu 0 vcpu total runtime 87399262934 number of ticks vcpu was idle 3940 timer interrupts generated by vlapic 10039 vcpu migration across host cpus 65 number of times hlt was ignored 0 total number of vm exits 8296713 root@olive:/vms # uname -a FreeBSD olive.example.com 10.0-CURRENT FreeBSD 10.0-CURRENT #8 r248949: Sun Mar 31 09:42:40 CDT 2013 root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 root@olive:/vms # top -bHI last pid: 1417; load averages: 1.96, 0.74, 0.35 up 0+00:26:00 10:38:13 23 processes: 3 running, 20 sleeping Mem: 21M Active, 13M Inact, 8166M Wired, 1920K Cache, 54G Free ARC: 832M Total, 310M MFU, 510M MRU, 8336K Anon, 2364K Header, 2125K Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1390 root 103 0 4120M 139M CPU7 7 1:48 100.00% bhyve{coco vcpu 0} 1390 root 102 0 4120M 139M CPU1 1 1:44 100.00% bhyve{coco vcpu 1} And here is the vm boot sequence and panic again which shows how the clock is going very fast (the crash only took about 2 minutes of wall time if that): FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Sun Mar 31 09:32:01 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2518 data=0x980e8+0xef8b0 syms=[0x8+0xca980+0x8+0x12bb86] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... ^PCopyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r248935: Sat Mar 30 13:56:05 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.83-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957530624 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/vtbd0p2 [rw]... WARNING: / was not properly dismounted WARNING: /: mount pending error: blocks 8 files 3 Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: ** SU+J Recovering /dev/vtbd0p2 ** Reading 33554432 byte journal from inode 4. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 89 journal records in 10240 bytes for 27.81% utilization ** Freed 9 inodes (0 dirs) 7 blocks, and 5 frags. ***** FILE SYSTEM MARKED CLEAN ***** g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: timed out waiting for child /etc/rc: WARNING: failed to start syslogd Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu May 9 19:33:49 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: panic: deadlkres: possible deadlock detected for 0xfffffe00a90a8000, blocked for 183132 ticks cpuid = 1 Uptime: 249d7h13m51s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 Consoles: userboot From owner-freebsd-virtualization@FreeBSD.ORG Mon Apr 1 11:06:53 2013 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 431E6912 for ; Mon, 1 Apr 2013 11:06:53 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 30BE734B for ; Mon, 1 Apr 2013 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r31B6rke033850 for ; Mon, 1 Apr 2013 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r31B6qRG033848 for freebsd-virtualization@FreeBSD.org; Mon, 1 Apr 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 1 Apr 2013 11:06:52 GMT Message-Id: <201304011106.r31B6qRG033848@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-virtualization@FreeBSD.org Subject: Current problem reports assigned to freebsd-virtualization@FreeBSD.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2013 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/170096 virtualization[vimage] Dynamically-attached network interface will c o kern/169991 virtualization[run] [vimage] panic after device plugged in o kern/165252 virtualization[vimage] [pf] [panic] kernel panics with VIMAGE and PF o kern/161094 virtualization[vimage] [pf] [panic] kernel panic with pf + VIMAGE wh o kern/160541 virtualization[vimage][pf][patch] panic: userret: Returning on td 0x o kern/160496 virtualization[vimage] [pf] [patch] kernel panic with pf + VIMAGE o kern/148155 virtualization[vimage] [pf] Kernel panic with PF/IPFilter + VIMAGE k a kern/147950 virtualization[vimage] [carp] VIMAGE + CARP = kernel crash s kern/143808 virtualization[pf] pf does not work inside jail a kern/141696 virtualization[rum] [vimage] [panic] rum(4)+ vimage = kernel panic 10 problems total. From owner-freebsd-virtualization@FreeBSD.ORG Mon Apr 1 19:01:56 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5AB53D4E; Mon, 1 Apr 2013 19:01:56 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 39BA79C5; Mon, 1 Apr 2013 19:01:56 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 82C57B94C; Mon, 1 Apr 2013 15:01:55 -0400 (EDT) From: John Baldwin To: freebsd-virtualization@freebsd.org Subject: Re: KVM with freeBSD and SR-IOV vlan doesn't working! Date: Mon, 1 Apr 2013 13:47:55 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <1364376687875-5799323.post@n5.nabble.com> In-Reply-To: <1364376687875-5799323.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201304011347.55406.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 01 Apr 2013 15:01:55 -0400 (EDT) Cc: freebsd-net@freebsd.org, kindule X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2013 19:01:56 -0000 On Wednesday, March 27, 2013 5:31:27 am kindule wrote: > Recently, we use KVM and SR-IOV in our project. And we choose freeBSD10 as > the guest os.As we use the ip address in the igb0 of our freeBSD10 guest, it > working no problem.However when i use vlan in our igb0 of the freeBSD10 > guest,we can see the vlan assigned right and we can ping the vlan address > without problem.But we add a gateway of the vlan area.we can't connnected to > the gateway. > there are some os messages: > Host: Debian 7.0 and KVM 1.2 > Guest: FreeBSD10-current > > And thanks for your help! Hmm, does the same vlan setup work on a standalone machine? -- John Baldwin From owner-freebsd-virtualization@FreeBSD.ORG Tue Apr 2 09:15:31 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7A3A0DB3 for ; Tue, 2 Apr 2013 09:15:31 +0000 (UTC) (envelope-from fivo.11235813@gmail.com) Received: from mail-wg0-f46.google.com (mail-wg0-f46.google.com [74.125.82.46]) by mx1.freebsd.org (Postfix) with ESMTP id 14E9A1B9 for ; Tue, 2 Apr 2013 09:15:30 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id l18so195414wgh.13 for ; Tue, 02 Apr 2013 02:15:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=QjPhTwlaYo31YFKFIo41SYXzW+cauNkcQGjJ/KGfo3w=; b=xugTN1pQ9N01mqehl0q9Hk/r1s58UzCUPSoWDmXsp5w2TWTiry0xvsrkD2LzTflHeT OeHvbyYoE8r/xBZHX2+1klWSZhb1k74BNeWfoosbHQXetUXKuUVy02eWibVnnAM6AeK/ pHkDwYl7U9r3cnwaTquN9ZOlbT7CafqjknGXijaTxLXFpLilD7lq9R0WBnK5odAyllOD a6ou4mlSzCYRyZAw2aFa4VUB5Xz1mBo3NeR8kHtnjFA2YNhdbU5K7R/tUE1y8djJh+Ms 5sGLikcgukBU7iZ+eMLzKsVdV+bXtKKkYQk9wFVf2girOrhpaP5/+jinmrSzmqx2S4Uq 9tzw== X-Received: by 10.180.182.36 with SMTP id eb4mr14235254wic.8.1364894130079; Tue, 02 Apr 2013 02:15:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.169.161 with HTTP; Tue, 2 Apr 2013 02:15:09 -0700 (PDT) In-Reply-To: References: From: =?UTF-8?Q?Iori=E3=80=80YONEJI?= Date: Tue, 2 Apr 2013 18:15:09 +0900 Message-ID: Subject: Re: suspend/resume on BHyVe To: Takuya ASADA Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Apr 2013 09:15:31 -0000 Hi all, I couldn't spare much time to consider about this topic because I have to move to new house, but now I'm OK. I'm implementing registers-save feature including: 1. libvmmapi with get_vmxctx 2. kernel supports of 1. 3. user-land hack using 1, that saves registers and some configurations into (possibly msgpack) binary file. But I noticed that vmxctx is depending on Intel VT-x, and we also need to support AMD SVM, so I need common struct registers. I heard about SVM support but it's not in my repository copy(Revision: 248994), how much do I have to care about AMD? From owner-freebsd-virtualization@FreeBSD.ORG Tue Apr 2 09:37:43 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 77DC5904 for ; Tue, 2 Apr 2013 09:37:43 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-pb0-f46.google.com (mail-pb0-f46.google.com [209.85.160.46]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5763F7 for ; Tue, 2 Apr 2013 09:37:43 +0000 (UTC) Received: by mail-pb0-f46.google.com with SMTP id rp8so144529pbb.33 for ; Tue, 02 Apr 2013 02:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dokukino.com; s=google; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=w3PIW5whRk1kAkDPRR96lAPGDXj9xglC03r3q+i1NtU=; b=dJNYSwkQjlU8xJBXX1N80oPxHyRUapwJXwPcVXLGCvqWa3nvtGml/jeacssf8roJkj Ild8Pv4ukzuyw6MdgXBoYPL1Ve7ErE6hkcQVzud7MogCwbl5vXFokbw9cEmpoqLxBQ4U OO8yw7A+y/lL4iwEhjvVy41uoDSlUUseCdEDA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:x-gm-message-state; bh=w3PIW5whRk1kAkDPRR96lAPGDXj9xglC03r3q+i1NtU=; b=pjWAO97kpLPLNkbHgLb36RiKPLHugp6TGh8I0mTJWGsEIPDwD3TVRr31974riu59k2 JlbDJI1aXefhYQDGAqsbs0JnZqpWXoEmlfMHotx8s2eBrPy6VQajqSx6hGVSzPiErToH v9H573gmX60OOVdbEgaBmkBMgUCt9qwEuQ0NC+LQQk4ztRT4oxeG961ARvgBeekgd6bK UAQi6UqVL1E2TlckDVRZ7TvZsp5aNixKELcbvMCoByntqSiRAq8bAOm85tvwMhUqubFU A+Zi7QxbXIpL1SwtirO/NOkEVDXMMZBA60VeeCK5j2vw+citXboDogM2Gt2gIQq9GDAw i/+Q== MIME-Version: 1.0 X-Received: by 10.66.19.201 with SMTP id h9mr23661304pae.188.1364895456982; Tue, 02 Apr 2013 02:37:36 -0700 (PDT) Received: by 10.68.243.41 with HTTP; Tue, 2 Apr 2013 02:37:36 -0700 (PDT) In-Reply-To: References: Date: Tue, 2 Apr 2013 18:37:36 +0900 Message-ID: Subject: Re: suspend/resume on BHyVe From: Takuya ASADA To: Iori YONEJI X-Gm-Message-State: ALoCoQkal0YwdI3in4Lz6ESdqE4BuaMWUJVhYBUKeLZE33dZ6+VI1kNDGfEPnvuUk8NRGDiCWp36 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Apr 2013 09:37:43 -0000 You can dump whole struct just a binary array(don't look into it in your program), and get sizeof(vmxctx) via ioctl return. It should enough to suspend/resume. Then if AMD SVM mode will have different values on the structure, your implementation won't break. Does it make sense? > Peter, Neel And, AMD SVM's vmxctx will not have much difference than Intel's one, I guess. (VMCS/VMCB layout are differ, though. But it's different topic.) 2013=E5=B9=B44=E6=9C=882=E6=97=A5=E7=81=AB=E6=9B=9C=E6=97=A5 Iori YONEJI fi= vo.11235813@gmail.com: > Hi all, > > I couldn't spare much time to consider about this topic because I have > to move to new house, but now I'm OK. > > I'm implementing registers-save feature including: > 1. libvmmapi with get_vmxctx > 2. kernel supports of 1. > 3. user-land hack using 1, that saves registers and some > configurations into (possibly msgpack) binary file. > > But I noticed that vmxctx is depending on Intel VT-x, and we also need > to support AMD SVM, > so I need common struct registers. > > I heard about SVM support but it's not in my repository copy(Revision: > 248994), > how much do I have to care about AMD? > From owner-freebsd-virtualization@FreeBSD.ORG Wed Apr 3 16:34:49 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8E8AA203 for ; Wed, 3 Apr 2013 16:34:49 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 4BB0BEC7 for ; Wed, 3 Apr 2013 16:34:49 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9417512299; Thu, 4 Apr 2013 02:34:41 +1000 (EST) Received: from vpn2ntap-731294.vpn.netapp.com (pos-ext.netapp.com [198.95.226.40]) by dommail.onthenet.com.au (MOS 4.2.4-GA) with ESMTP id BLE29008 (AUTH peterg@ptree32.com.au); Thu, 4 Apr 2013 02:34:39 +1000 Message-ID: <515C5A1B.5070903@freebsd.org> Date: Wed, 03 Apr 2013 10:34:35 -0600 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 To: Takuya ASADA Subject: Re: suspend/resume on BHyVe References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2013 16:34:49 -0000 > You can dump whole struct just a binary array(don't look into it in your > program), and get sizeof(vmxctx) via ioctl return. It should enough to > suspend/resume. > Then if AMD SVM mode will have different values on the structure, your > implementation won't break. > > Does it make sense? > Peter, Neel I was thinking of something a bit more abstract. There shouldn't be any need to encode VMCS/SVM info in the save/restore state - that can be recreated on demand. For instance, bhyveload can pass off execution state to bhyve without having any knowledge of VMCS/SVM. You'd need to represent the entire CPU register state of the VM, memory, and also enough for the virtual devices to be able to resurrect state. Plus, there needs to be some versioning to allow suspend/resume between different revs of bhyve. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Sat Apr 6 01:27:36 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 122D5D8B for ; Sat, 6 Apr 2013 01:27:36 +0000 (UTC) (envelope-from torek@torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id E1BED69E for ; Sat, 6 Apr 2013 01:27:34 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r361RRGv068417 for ; Fri, 5 Apr 2013 19:27:27 -0600 (MDT) (envelope-from torek@torek.net) Message-Id: <201304060127.r361RRGv068417@elf.torek.net> From: Chris Torek To: freebsd-virtualization@freebsd.org Subject: two more small improvements Date: Fri, 05 Apr 2013 19:27:27 -0600 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (elf.torek.net [127.0.0.1]); Fri, 05 Apr 2013 19:27:27 -0600 (MDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2013 01:27:36 -0000 When I first went to run bhyve on one hardware box I had the entire system crash. This turned out to be due to the BIOS disabling VMX. A message came out on the console, but the scripts continued anyway, and then, boom. I first looked at the wrong bit of code for fixing this, but I *think* I improved it anyway. :-) That's the first patch below. The second patch below (note: I'll put them somewhere on torek.net if that's better than just including them here) fixes the crash. The easiest way to reproduce it is to attempt to create a VM within a bhyve VM ("bh1" is a running VM): [before patch -- note, some wraparound fixed] root@bh1:~ # kldload vmm vmx_init: processor does not support VMX operation module_register_init: MOD_LOAD (vmm, 0xffffffff81a133b0, 0) error 6 root@bh1:~ # sysctl hw.vmm hw.vmm.destroy: beavis hw.vmm.create: beavis hw.vmm.pages_allocated: 0 root@bh1:~ # sysctl hw.vmm.create hw.vmm.create: beavis root@bh1:~ # sysctl hw.vmm.create= hw.vmm.create: beavisvm exit[1] reason VMX rip 0xffffffff81a1932c inst_length 6 error 0 exit_reason 50 qualification 0xfffffffffffffff0 vm exit[0] reason VMX rip 0xffffffff81a1932c inst_length 6 error 0 exit_reason 50 qualification 0xfffffffffffffff0 [after patch] root@bh1:~ # kldload vmm vmx_init: processor does not support VMX operation module_register_init: MOD_LOAD (vmm, 0xffffffff81a133d0, 0) error 6 root@bh1:~ # sysctl hw.vmm hw.vmm.destroy: beavis hw.vmm.create: beavis hw.vmm.pages_allocated: 0 root@bh1:~ # sysctl hw.vmm.create hw.vmm.create: beavis root@bh1:~ # sysctl hw.vmm.create= hw.vmm.create: beavis sysctl: hw.vmm.create=: Device not configured root@bh1:~ # Chris changeset: 1962:7e204f321854 user: Chris Torek date: Fri Apr 05 19:10:09 2013 -0600 files: sys/amd64/vmm/intel/vmx.c description: improve test for VMX-mode We only need VMX extensions to be enabled, not locked. Moreover, we should not care about any VMX-in-SMX settings. diff --git a/sys/amd64/vmm/intel/vmx.c b/sys/amd64/vmm/intel/vmx.c --- a/sys/amd64/vmm/intel/vmx.c +++ b/sys/amd64/vmm/intel/vmx.c @@ -436,14 +436,33 @@ return (ENXIO); } +/* + * Bits in MSR_IA32_FEATURE_CONTROL register. + * + * XXX move these to machine/specialreg.h + */ +#define MSR_IA32_FEAT_CTL_LOCK 0x01 /* locks the control reg */ +#define MSR_IA32_FEAT_CTL_VMX_SMX_EN 0x02 /* enable VMXON in SMX mode */ +#define MSR_IA32_FEAT_CTL_VMX_EN 0x04 /* enable VMXON (non-SMX) */ /* - * Verify that MSR_IA32_FEATURE_CONTROL lock and VMXON enable bits - * are set (bits 0 and 2 respectively). + * Verify that VMXON is allowed. + * + * According to Intel docs, we just need VMX_EN to be + * set. If the LOCK bit is not set we can set or clear + * VMX_EN ourselves. Once the LOCK bit is set no more + * changes are possible without a processor reset. + * + * Existing BIOSes currently set-and-lock the feature, but + * this code should work with BIOSes that don't lock it. */ feature_control = rdmsr(MSR_IA32_FEATURE_CONTROL); - if ((feature_control & 0x5) != 0x5) { - printf("vmx_init: VMX operation disabled by BIOS\n"); - return (ENXIO); + if ((feature_control & MSR_IA32_FEAT_CTL_VMX_EN) == 0) { + if (feature_control & MSR_IA32_FEAT_CTL_LOCK) { + printf("vmx_init: VMX operation disabled by BIOS\n"); + return (ENXIO); + } + feature_control |= MSR_IA32_FEAT_CTL_VMX_EN; + wrmsr(MSR_IA32_FEATURE_CONTROL, feature_control); } /* Check support for primary processor-based VM-execution controls */ changeset: 1963:6cea2a2ed727 tag: tip user: Chris Torek date: Fri Apr 05 19:14:05 2013 -0600 files: sys/amd64/include/vmm.h sys/amd64/vmm/vmm.c sys/amd64/vmm/vmm_dev.c description: prevent host OS crash when VMX is disabled If VMX is disabled in the BIOS, vmm_init correctly returns an error, but this still leaves the module loaded and running, and a later sysctl to create a virtual machine crashed the host. Have vm_create() check, and return an error for this condition. diff --git a/sys/amd64/include/vmm.h b/sys/amd64/include/vmm.h --- a/sys/amd64/include/vmm.h +++ b/sys/amd64/include/vmm.h @@ -87,7 +87,7 @@ extern struct vmm_ops vmm_ops_intel; extern struct vmm_ops vmm_ops_amd; -struct vm *vm_create(const char *name); +int vm_create(const char *name, struct vm **retval); void vm_destroy(struct vm *vm); const char *vm_name(struct vm *vm); int vm_malloc(struct vm *vm, vm_paddr_t gpa, size_t len); diff --git a/sys/amd64/vmm/vmm.c b/sys/amd64/vmm/vmm.c --- a/sys/amd64/vmm/vmm.c +++ b/sys/amd64/vmm/vmm.c @@ -213,6 +213,15 @@ vmmdev_init(); iommu_init(); error = vmm_init(); + if (error) { + /* + * Returning an error here does not force + * the module to be unloaded, so we have + * to make sure that vm_create()s will not + * proceed. + */ + ops = NULL; + } break; case MOD_UNLOAD: error = vmmdev_cleanup(); @@ -249,8 +258,8 @@ SYSCTL_NODE(_hw, OID_AUTO, vmm, CTLFLAG_RW, NULL, NULL); -struct vm * -vm_create(const char *name) +int +vm_create(const char *name, struct vm **retval) { int i; struct vm *vm; @@ -258,8 +267,11 @@ const int BSP = 0; + if (ops == NULL) + return (ENXIO); + if (name == NULL || strlen(name) >= VM_MAX_NAMELEN) - return (NULL); + return (EINVAL); vm = malloc(sizeof(struct vm), M_VM, M_WAITOK | M_ZERO); strcpy(vm->name, name); @@ -274,7 +286,8 @@ vm->iommu = iommu_create_domain(maxaddr); vm_activate_cpu(vm, BSP); - return (vm); + *retval = vm; + return (0); } static void diff --git a/sys/amd64/vmm/vmm_dev.c b/sys/amd64/vmm/vmm_dev.c --- a/sys/amd64/vmm/vmm_dev.c +++ b/sys/amd64/vmm/vmm_dev.c @@ -475,9 +475,9 @@ if (sc != NULL) return (EEXIST); - vm = vm_create(buf); - if (vm == NULL) - return (EINVAL); + error = vm_create(buf, &vm); + if (error) + return (error); sc = malloc(sizeof(struct vmmdev_softc), M_VMMDEV, M_WAITOK | M_ZERO); sc->vm = vm;