Date: Tue, 9 Apr 2013 23:05:44 -0700 From: Neel Natu <neelnatu@gmail.com> To: Dan Mack <mack@macktronics.com> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: bhyve tty / login problems / panic Message-ID: <CAFgRE9Hv6aHbqp0c7BS=siO%2Br8sQGBPSkHYUAi%2B6_MLZHb97gQ@mail.gmail.com> In-Reply-To: <CAFgRE9FWoAEqNm3Aai4gkw6khRjDZ03BP_kQZNosLm4jb-r=0A@mail.gmail.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <CAFgRE9HiHHEgZ4u%2BNd_7tOMd342XiV9mu8pSt4fYjJhGvowQ5Q@mail.gmail.com> <20130329172023.L15559@coco.macktronics.com> <CAFgRE9G6mND6s_D3=3UznegtiXZoZtrma_jyCgLGbZ1-5Kcpjg@mail.gmail.com> <20130330125954.G17063@coco.macktronics.com> <20130330141141.B17063@coco.macktronics.com> <CAFgRE9FWoAEqNm3Aai4gkw6khRjDZ03BP_kQZNosLm4jb-r=0A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, We did some more debugging on Dan's system and root caused the issue to unsynchronized TSCs on the host system. A fix has been committed for this issue: http://svnweb.freebsd.org/base?view=revision&revision=249324 best Neel On Sat, Mar 30, 2013 at 9:03 PM, Neel Natu <neelnatu@gmail.com> wrote: > Hi Dan, > > On Sat, Mar 30, 2013 at 12:17 PM, Dan Mack <mack@macktronics.com> wrote: > >> Here you go: >> >> - uni-processor output: >> >> >> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >> vcpu0 >> vm exits due to external interrupt 1330 >> number of times hlt was intercepted 676 >> number of times %cr access was intercepted 1 >> number of times rdmsr was intercepted 0 >> number of times wrmsr was intercepted 4 >> number of monitor trap exits 0 >> number of times pause was intercepted 428499 >> vm exits due to interrupt window opening 1231 >> vm exits due to nmi window opening 0 >> number of times in/out was intercepted 423985 >> number of times cpuid was intercepted 22 >> vm exits due to nested page fault 7523 >> number of vm exits for unknown reason 0 >> number of times astpending at exit 38 >> number of vm exits handled in userspace 853290 >> >> number of NMIs delivered to vcpu 0 >> vcpu total runtime 6954314530 >> number of ticks vcpu was idle 59572 >> vcpu migration across host cpus 29 >> >> number of times hlt was ignored 0 >> total number of vm exits 863271 >> >> And here is when running with vcpu==2: >> >> >> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >> vcpu0 >> vm exits due to external interrupt 19234 >> number of times hlt was intercepted 1965 >> number of times %cr access was intercepted 1 >> number of times rdmsr was intercepted 0 >> number of times wrmsr was intercepted 4 >> number of monitor trap exits 0 >> number of times pause was intercepted 3907709 >> vm exits due to interrupt window opening 8105 >> vm exits due to nmi window opening 0 >> number of times in/out was intercepted 457998 >> number of times cpuid was intercepted 50 >> vm exits due to nested page fault 940909 >> number of vm exits for unknown reason 0 >> number of times astpending at exit 102 >> number of vm exits handled in userspace 4367825 >> >> number of NMIs delivered to vcpu 0 >> vcpu total runtime 51834598990 >> number of ticks vcpu was idle 887 >> vcpu migration across host cpus 25 >> >> number of times hlt was ignored 0 >> total number of vm exits 5335975 >> >> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >> vcpu1 >> vm exits due to external interrupt 51249 >> number of times hlt was intercepted 3676 >> number of times %cr access was intercepted 2 >> number of times rdmsr was intercepted 0 >> number of times wrmsr was intercepted 4 >> number of monitor trap exits 0 >> number of times pause was intercepted 4667527 >> vm exits due to interrupt window opening 32453 >> vm exits due to nmi window opening 0 >> number of times in/out was intercepted 57278 >> number of times cpuid was intercepted 63 >> vm exits due to nested page fault 266091 >> number of vm exits for unknown reason 0 >> number of times astpending at exit 164 >> number of vm exits handled in userspace 4728481 >> >> number of NMIs delivered to vcpu 0 >> vcpu total runtime 53833466076 >> number of ticks vcpu was idle 965 >> vcpu migration across host cpus 16 >> >> number of times hlt was ignored 0 >> total number of vm exits 5078343 >> >> The panic comes eventually like this: >> >> login: panic: deadlkres: possible deadlock detected for >> 0xfffffe0090fb5490, blocked for 217798 ticks >> >> cpuid = 0 >> Uptime: 255d6h29m56s >> Automatic reboot in 15 seconds - press a key on the console to abort >> >> > Thanks for capturing that. > > In the SMP case the aberrant source of VM exits is the nested page table > faults - which means local apic accesses since these are being entirely > handled in the kernel. They are an order of magnitude more than in the UP > case - even after adjusting for the difference in the run times in the two > cases. > > Also, the uptime message (255d6h29m56s) is indicates something funky > happening with the time keeping. > > I have added one more stat to vmm.ko that counts the number of times a > local apic timer interrupt is generated for each vcpu. > > Also, I would like to install your release.iso if you are able to upload > it. I very much doubt, it but want to rule out those specific bits as the > source of the problem. > > Thanks again for helping to debug this. > > best > Neel > > >> >> Hope this helps, and FYI, here's a dmesg from the host: >> >> >> Copyright (c) 1992-2013 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013 >> root@olive.example.com:/usr/**obj/usr/src/sys/MACKGEN amd64 >> >> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU) >> >> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >> Stepping = 7 >> >> Features=0xbfebfbff<FPU,VME,**DE,PSE,TSC,MSR,PAE,MCE,CX8,** >> APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** >> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> >> >> Features2=0x1fbee3bf<SSE3,**PCLMULQDQ,DTES64,MON,DS_CPL,** >> VMX,EST,TM2,SSSE3,CX16,xTPR,**PDCM,PCID,DCA,SSE4.1,SSE4.2,** >> x2APIC,POPCNT,TSCDLT,AESNI,**XSAVE,OSXSAVE,AVX> >> AMD Features=0x2c100800<SYSCALL,**NX,Page1GB,RDTSCP,LM> >> AMD Features2=0x1<LAHF> >> TSC: P-state invariant, performance statistics >> real memory = 68719476736 (65536 MB) >> avail memory = 63123800064 (60199 MB) >> Event timer "LAPIC" quality 600 >> ACPI APIC Table: <INTEL DX79SI > >> FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs >> FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads >> >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 2 >> cpu3 (AP): APIC ID: 3 >> cpu4 (AP): APIC ID: 4 >> cpu5 (AP): APIC ID: 5 >> cpu6 (AP): APIC ID: 6 >> cpu7 (AP): APIC ID: 7 >> cpu8 (AP): APIC ID: 8 >> cpu9 (AP): APIC ID: 9 >> cpu10 (AP): APIC ID: 10 >> cpu11 (AP): APIC ID: 11 >> ioapic1 <Version 2.0> irqs 24-47 on motherboard >> ioapic0 <Version 2.0> irqs 0-23 on motherboard >> lapic0: Forcing LINT1 to edge trigger >> kbd1 at kbdmux0 >> acpi0: <INTEL DX79SI> on motherboard >> acpi0: Power Button (fixed) >> acpi0: reservation of f0000000, 18000000 (3) failed >> cpu0: <ACPI CPU> on acpi0 >> cpu1: <ACPI CPU> on acpi0 >> cpu2: <ACPI CPU> on acpi0 >> cpu3: <ACPI CPU> on acpi0 >> cpu4: <ACPI CPU> on acpi0 >> cpu5: <ACPI CPU> on acpi0 >> cpu6: <ACPI CPU> on acpi0 >> cpu7: <ACPI CPU> on acpi0 >> cpu8: <ACPI CPU> on acpi0 >> cpu9: <ACPI CPU> on acpi0 >> cpu10: <ACPI CPU> on acpi0 >> cpu11: <ACPI CPU> on acpi0 >> atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0 >> >> Event timer "RTC" frequency 32768 Hz quality 0 >> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> Event timer "i8254" frequency 1193182 Hz quality 100 >> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed03fff on acpi0 >> Timecounter "HPET" frequency 14318180 Hz quality 950 >> Event timer "HPET" frequency 14318180 Hz quality 350 >> Event timer "HPET1" frequency 14318180 Hz quality 340 >> Event timer "HPET2" frequency 14318180 Hz quality 340 >> Event timer "HPET3" frequency 14318180 Hz quality 340 >> Event timer "HPET4" frequency 14318180 Hz quality 340 >> Event timer "HPET5" frequency 14318180 Hz quality 340 >> Event timer "HPET6" frequency 14318180 Hz quality 340 >> Event timer "HPET7" frequency 14318180 Hz quality 340 >> >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> acpi_button0: <Sleep Button> on acpi0 >> >> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 >> pci0: <ACPI PCI bus> on pcib0 >> pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 >> pci1: <ACPI PCI bus> on pcib1 >> pcib2: <ACPI PCI-PCI bridge> at device 1.1 on pci0 >> pci2: <ACPI PCI bus> on pcib2 >> xhci0: <NEC uPD720200 USB 3.0 controller> mem 0xe3400000-0xe3401fff irq >> 16 at device 0.0 on pci2 >> usbus0: waiting for BIOS to give up control >> xhci0: 32 byte context size. >> usbus0 on xhci0 >> pcib3: <ACPI PCI-PCI bridge> at device 2.0 on pci0 >> pci3: <ACPI PCI bus> on pcib3 >> vgapci0: <VGA-compatible display> port 0x3000-0x307f mem >> 0xe2000000-0xe2ffffff,**0xd0000000-0xdfffffff,**0xe0000000-0xe1ffffff >> irq 16 at device 0.0 on pci3 >> pci3: <multimedia, HDA> at device 0.1 (no driver attached) >> pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0 >> pci4: <ACPI PCI bus> on pcib4 >> pci0: <base peripheral> at device 5.0 (no driver attached) >> pci0: <base peripheral> at device 5.2 (no driver attached) >> pcib5: <ACPI PCI-PCI bridge> at device 17.0 on pci0 >> pci5: <ACPI PCI bus> on pcib5 >> pci0: <simple comms> at device 22.0 (no driver attached) >> em0: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x4040-0x405f mem >> 0xe3500000-0xe351ffff,**0xe3521000-0xe3521fff irq 20 at device 25.0 on >> pci0 >> em0: Using an MSI interrupt >> em0: Ethernet address: 4c:72:b9:d1:b2:d2 >> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xe3601000-0xe36013ff irq >> 16 at device 26.0 on pci0 >> usbus1: EHCI version 1.0 >> usbus1 on ehci0 >> pcib6: <ACPI PCI-PCI bridge> at device 28.0 on pci0 >> pci6: <ACPI PCI bus> on pcib6 >> pcib7: <ACPI PCI-PCI bridge> at device 28.2 on pci0 >> pci7: <ACPI PCI bus> on pcib7 >> pci7: <serial bus, FireWire> at device 0.0 (no driver attached) >> pcib8: <ACPI PCI-PCI bridge> at device 28.3 on pci0 >> pci8: <ACPI PCI bus> on pcib8 >> xhci1: <NEC uPD720200 USB 3.0 controller> mem 0xe3200000-0xe3201fff irq >> 19 at device 0.0 on pci8 >> usbus2: waiting for BIOS to give up control >> xhci1: 32 byte context size. >> usbus2 on xhci1 >> pcib9: <ACPI PCI-PCI bridge> at device 28.5 on pci0 >> pci9: <ACPI PCI bus> on pcib9 >> em1: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x1000-0x101f mem >> 0xe3100000-0xe311ffff,**0xe3120000-0xe3123fff irq 17 at device 0.0 on >> pci9 >> em1: Using MSIX interrupts with 3 vectors >> em1: Ethernet address: 4c:72:b9:d1:b2:d3 >> ehci1: <EHCI (generic) USB 2.0 controller> mem 0xe3602000-0xe36023ff irq >> 23 at device 29.0 on pci0 >> usbus3: EHCI version 1.0 >> usbus3 on ehci1 >> pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0 >> pci10: <ACPI PCI bus> on pcib10 >> isab0: <PCI-ISA bridge> at device 31.0 on pci0 >> isa0: <ISA bus> on isab0 >> ahci0: <Intel Patsburg AHCI SATA controller> port >> 0x4068-0x406f,0x4074-0x4077,**0x4060-0x4067,0x4070-0x4073,**0x4020-0x403f >> mem 0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0 >> ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported >> ahcich0: <AHCI channel> at channel 0 on ahci0 >> ahcich1: <AHCI channel> at channel 1 on ahci0 >> ahcich2: <AHCI channel> at channel 2 on ahci0 >> ahcich3: <AHCI channel> at channel 3 on ahci0 >> ahcich4: <AHCI channel> at channel 4 on ahci0 >> ahcich5: <AHCI channel> at channel 5 on ahci0 >> ahciem0: <AHCI enclosure management bridge> on ahci0 >> pci0: <serial bus, SMBus> at device 31.3 (no driver attached) >> orm0: <ISA Option ROMs> at iomem 0xd1000-0xd1fff,0xd2000-**0xd2fff on >> isa0 >> sc0: <System console> at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=0x300> >> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 >> atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 >> atkbd0: <AT Keyboard> irq 1 on atkbdc0 >> kbd0 at atkbd0 >> atkbd0: [GIANT-LOCKED] >> ppc0: cannot reserve I/O port range >> est0: <Enhanced SpeedStep Frequency Control> on cpu0 >> p4tcc0: <CPU Frequency Thermal Control> on cpu0 >> est1: <Enhanced SpeedStep Frequency Control> on cpu1 >> p4tcc1: <CPU Frequency Thermal Control> on cpu1 >> est2: <Enhanced SpeedStep Frequency Control> on cpu2 >> p4tcc2: <CPU Frequency Thermal Control> on cpu2 >> est3: <Enhanced SpeedStep Frequency Control> on cpu3 >> p4tcc3: <CPU Frequency Thermal Control> on cpu3 >> est4: <Enhanced SpeedStep Frequency Control> on cpu4 >> p4tcc4: <CPU Frequency Thermal Control> on cpu4 >> est5: <Enhanced SpeedStep Frequency Control> on cpu5 >> p4tcc5: <CPU Frequency Thermal Control> on cpu5 >> est6: <Enhanced SpeedStep Frequency Control> on cpu6 >> p4tcc6: <CPU Frequency Thermal Control> on cpu6 >> est7: <Enhanced SpeedStep Frequency Control> on cpu7 >> p4tcc7: <CPU Frequency Thermal Control> on cpu7 >> est8: <Enhanced SpeedStep Frequency Control> on cpu8 >> p4tcc8: <CPU Frequency Thermal Control> on cpu8 >> est9: <Enhanced SpeedStep Frequency Control> on cpu9 >> p4tcc9: <CPU Frequency Thermal Control> on cpu9 >> est10: <Enhanced SpeedStep Frequency Control> on cpu10 >> p4tcc10: <CPU Frequency Thermal Control> on cpu10 >> est11: <Enhanced SpeedStep Frequency Control> on cpu11 >> p4tcc11: <CPU Frequency Thermal Control> on cpu11 >> ZFS filesystem version: 5 >> ZFS storage pool version: features support (5000) >> Timecounters tick every 1.000 msec >> usbus0: 5.0Gbps Super Speed USB v3.0 >> usbus1: 480Mbps High Speed USB v2.0 >> usbus2: 5.0Gbps Super Speed USB v3.0 >> usbus3: 480Mbps High Speed USB v2.0 >> ugen1.1: <Intel> at usbus1 >> uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 >> ugen0.1: <0x1033> at usbus0 >> uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 >> ugen3.1: <Intel> at usbus3 >> uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 >> ugen2.1: <0x1033> at usbus2 >> uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 >> ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 >> ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device >> ses0: SEMB SES Device >> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 >> ada0: <OCZ-VERTEX4 1.5> ATA-9 SATA 3.x device >> ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) >> ada0: Command Queueing enabled >> ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C) >> ada0: Previously was known as ad4 >> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 >> ada1: <INTEL SSDSC2CW120A3 400i> ATA-9 SATA 3.x device >> ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) >> ada1: Command Queueing enabled >> ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) >> ada1: Previously was known as ad6 >> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 >> ada2: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device >> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada2: Command Queueing enabled >> ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada2: Previously was known as ad8 >> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 >> ada3: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device >> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada3: Command Queueing enabled >> ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada3: Previously was known as ad10 >> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 >> ada4: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device >> ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada4: Command Queueing enabled >> ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada4: Previously was known as ad12 >> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 >> ada5: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device >> ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada5: Command Queueing enabled >> ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) >> ada5: Previously was known as ad14 >> lapic1: Forcing LINT1 to edge trigger >> >> SMP: AP CPU #1 Launched! >> lapic4: Forcing LINT1 to edge trigger >> SMP: AP CPU #4 Launched! >> lapic11: Forcing LINT1 to edge trigger >> SMP: AP CPU #11 Launched! >> lapic7: Forcing LINT1 to edge trigger >> SMP: AP CPU #7 Launched! >> lapic5: Forcing LINT1 to edge trigger >> SMP: AP CPU #5 Launched! >> lapic9: Forcing LINT1 to edge trigger >> SMP: AP CPU #9 Launched! >> lapic2: Forcing LINT1 to edge trigger >> SMP: AP CPU #2 Launched! >> lapic3: Forcing LINT1 to edge trigger >> SMP: AP CPU #3 Launched! >> lapic10: Forcing LINT1 to edge trigger >> SMP: AP CPU #10 Launched! >> lapic6: Forcing LINT1 to edge trigger >> SMP: AP CPU #6 Launched! >> lapic8: Forcing LINT1 to edge trigger >> SMP: AP CPU #8 Launched! >> uhub3: 4 ports with 4 removable, self powered >> uhub1: 4 ports with 4 removable, self powered >> Root mount waiting for: usbus3 usbus1 >> uhub0: 2 ports with 2 removable, self powered >> uhub2: 2 ports with 2 removable, self powered >> Root mount waiting for: usbus3 usbus1 >> ugen1.2: <vendor 0x8087> at usbus1 >> uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> >> on usbus1 >> ugen3.2: <vendor 0x8087> at usbus3 >> uhub5: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> >> on usbus3 >> Root mount waiting for: usbus3 usbus1 >> uhub4: 6 ports with 6 removable, self powered >> uhub5: 8 ports with 8 removable, self powered >> ugen1.3: <vendor 0x060b> at usbus1 >> ukbd0: <vendor 0x060b USB Keyboard, class 0/0, rev 2.00/1.10, addr 3> on >> usbus1 >> kbd2 at ukbd0 >> ugen3.3: <Ralink> at usbus3 >> Root mount waiting for: usbus3 >> ugen3.4: <vendor 0x0db0> at usbus3 >> Trying to mount root from zfs:tron []... >> ums0: <vendor 0x060b USB Keyboard, class 0/0, rev 2.00/1.10, addr 3> on >> usbus1 >> ums0: 5 buttons and [XYZ] coordinates ID=0 >> run0: <1.0> on usbus3 >> run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address >> 94:db:c9:e3:6e:c8 >> ubt0: <vendor 0x0db0 product 0xa871, class 224/1, rev 2.00/52.76, addr 4> >> on usbus3 >> WARNING: attempt to domain_add(bluetooth) after domainfinalize() >> WARNING: attempt to domain_add(netgraph) after domainfinalize() >> tap0: Ethernet address: 00:bd:8c:54:07:00 >> tap0: link state changed to UP >> bridge0: Ethernet address: 02:4c:89:ce:33:00 >> tap0: promiscuous mode enabled >> bridge0: link state changed to UP >> em1: link state changed to DOWN >> em1: promiscuous mode enabled >> em1: link state changed to UP >> tap0: link state changed to DOWN >> tap0: link state changed to UP >> tap0: link state changed to DOWN >> tap0: link state changed to UP >> >> >> >> >>> Yep, no problem. I saw the checkin and started a rebuild a few minutes >>> ago :-) >>> >>> Thanks, >>> >>> Dan >>> >>> On Sat, 30 Mar 2013, Neel Natu wrote: >>> >>> Hi Dan, >>>> >>>> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack <mack@macktronics.com> wrote: >>>> >>>> >>>>> >>>>> Here you go Neel: >>>>> >>>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >>>>> vcpu0 >>>>> vm exits due to external interrupt 45051 >>>>> number of times hlt was intercepted 2277 >>>>> number of NMIs delivered to vcpu 0 >>>>> vcpu total runtime 105001652482 >>>>> number of ticks vcpu was idle 919 >>>>> vcpu migration across host cpus 13 >>>>> number of times hlt was ignored 0 >>>>> total number of vm exits 9240925 >>>>> root@olive:~ # >>>>> root@olive:~ # >>>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >>>>> vcpu1 >>>>> vm exits due to external interrupt 149431 >>>>> number of times hlt was intercepted 4222 >>>>> number of NMIs delivered to vcpu 0 >>>>> vcpu total runtime 106876622528 >>>>> number of ticks vcpu was idle 869 >>>>> vcpu migration across host cpus 20 >>>>> number of times hlt was ignored 0 >>>>> total number of vm exits 8065909 >>>>> >>>>> >>>>> That did not help a whole lot because we were not keeping track of >>>> all the >>>> reasons a vcpu could exit. >>>> >>>> I have fixed that in r248935: >>>> http://svnweb.freebsd.org/**base?view=revision&revision=**248935<http://svnweb.freebsd.org/base?view=revision&revision=248935>; >>>> >>>> Do you mind updating your vmm.ko with the change and getting the stats >>>> again? >>>> >>>> best >>>> Neel >>>> >>>> >>>> Dan >>>>> >>>>> >>>>> >>>>> On Fri, 29 Mar 2013, Neel Natu wrote: >>>>> >>>>> Hi Dan, >>>>> >>>>>> >>>>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack <mack@macktronics.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>> I ran the procsystime dtrace script on bhyve when it was chewing up >>>>>>> all >>>>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of >>>>>>> runtime: >>>>>>> >>>>>>> root@olive:/usr/share/dtrace/******toolkit # ./procsystime -n bhyve >>>>>>> -aT >>>>>>> >>>>>>> Tracing... Hit Ctrl-C to end... >>>>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>>>>> ^C >>>>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>>>>> >>>>>>> Elapsed Times for processes bhyve, >>>>>>> >>>>>>> SYSCALL TIME (ns) >>>>>>> _umtx_op 18349 >>>>>>> writev 135712 >>>>>>> preadv 16175267 >>>>>>> pwritev 22924378 >>>>>>> ioctl 4353897920 >>>>>>> TOTAL: 4393151626 >>>>>>> >>>>>>> CPU Times for processes bhyve, >>>>>>> >>>>>>> SYSCALL TIME (ns) >>>>>>> _umtx_op 8815 >>>>>>> writev 103145 >>>>>>> pwritev 10647023 >>>>>>> preadv 15159976 >>>>>>> ioctl 3943399889 >>>>>>> TOTAL: 3969318848 >>>>>>> >>>>>>> Syscall Counts for processes bhyve, >>>>>>> >>>>>>> SYSCALL COUNT >>>>>>> _umtx_op 17 >>>>>>> writev 43 >>>>>>> pwritev 639 >>>>>>> preadv 1183 >>>>>>> ioctl 652782 >>>>>>> TOTAL: 654664 >>>>>>> >>>>>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>>>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> root@olive:/usr/share/dtrace/******toolkit # ./hotkernel >>>>>>> >>>>>>> Sampling... Hit Ctrl-C to end. >>>>>>> ^C >>>>>>> FUNCTION COUNT PCNT >>>>>>> zfs.ko`space_map_sync 1 0.0% >>>>>>> kernel`syscall_thread_exit 1 0.0% >>>>>>> kernel`should_yield 1 0.0% >>>>>>> zfs.ko`lzjb_compress 1 0.0% >>>>>>> zfs.ko`dbuf_rele 1 0.0% >>>>>>> zfs.ko`dsl_pool_tempreserve_******space 1 >>>>>>> 0.0% >>>>>>> >>>>>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>>>>> kernel`cpu_idleclock 1 0.0% >>>>>>> kernel`_rw_runlock_cookie 1 0.0% >>>>>>> kernel`pmap_enter 1 0.0% >>>>>>> kernel`__lockmgr_args 1 0.0% >>>>>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>>>>> kernel`ahci_ch_intr 1 0.0% >>>>>>> kernel`__rw_rlock 1 0.0% >>>>>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>>>>> zfs.ko`arc_buf_thaw 1 0.0% >>>>>>> kernel`atomic_set_int 1 0.0% >>>>>>> kernel`g_io_schedule_down 1 0.0% >>>>>>> kernel`__mtx_lock_flags 1 0.0% >>>>>>> kernel`_callout_stop_safe 1 0.0% >>>>>>> kernel`sched_idletd 1 0.0% >>>>>>> kernel`_sx_slock 1 0.0% >>>>>>> vmm.ko`lapic_set_intr 1 0.0% >>>>>>> kernel`cpusetobj_ffs 1 0.0% >>>>>>> zfs.ko`dbuf_hold 1 0.0% >>>>>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>>>>> zfs.ko`sa_attr_op 1 0.0% >>>>>>> zfs.ko`dmu_buf_hold_array_by_******dnode 1 >>>>>>> 0.0% >>>>>>> >>>>>>> zfs.ko`taskq_run_safe 1 0.0% >>>>>>> kernel`vm_reserv_free_page 1 0.0% >>>>>>> kernel`_sx_sunlock 1 0.0% >>>>>>> kernel`vn_rlimit_fsize 1 0.0% >>>>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>>>>> kernel`vn_write 1 0.0% >>>>>>> kernel`lock_init 1 0.0% >>>>>>> vmm.ko`vmx_resume 1 0.0% >>>>>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>>>>> kernel`mnt_vnode_next_active 1 0.0% >>>>>>> kernel`vm_page_remove 1 0.0% >>>>>>> zfs.ko`vdev_queue_deadline_******compare 1 >>>>>>> 0.0% >>>>>>> >>>>>>> kernel`zone_free_item 1 0.0% >>>>>>> vmm.ko`vmx_launch 1 0.0% >>>>>>> kernel`m_pullup 1 0.0% >>>>>>> kernel`pmap_change_wiring 1 0.0% >>>>>>> kernel`sched_userret 1 0.0% >>>>>>> zfs.ko`dbuf_write_done 1 0.0% >>>>>>> zfs.ko`zio_execute 1 0.0% >>>>>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>>>>> zfs.ko`avl_rotation 2 0.0% >>>>>>> kernel`ehci_device_isoc_fs_******enter 2 >>>>>>> 0.0% >>>>>>> >>>>>>> vmm.ko`vm_exitinfo 2 0.0% >>>>>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>>>>> vmm.ko`vcpu_stats 2 0.0% >>>>>>> kernel`atomic_add_long 2 0.0% >>>>>>> vmm.ko`0xffffffff81 2 0.0% >>>>>>> kernel`vm_reserv_alloc_page 2 0.0% >>>>>>> kernel`pagezero 2 0.0% >>>>>>> kernel`bzero 2 0.0% >>>>>>> zfs.ko`fletcher_2_native 2 0.0% >>>>>>> vmm.ko`vmx_getreg 2 0.0% >>>>>>> vmm.ko`vm_guest_msrs 3 0.0% >>>>>>> vmm.ko`vmx_setreg 3 0.0% >>>>>>> vmm.ko`vm_get_register 3 0.0% >>>>>>> kernel`sx_try_xlock_ 3 0.0% >>>>>>> kernel`vm_radix_lookup_le 3 0.0% >>>>>>> vmm.ko`lapic_pending_intr 4 0.0% >>>>>>> kernel`syscall_thread_enter 4 0.0% >>>>>>> kernel`cpu_set_syscall_retval 4 0.0% >>>>>>> kernel`__mnt_vnode_next_all 4 0.0% >>>>>>> kernel`vfs_stdsync 4 0.0% >>>>>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>>>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>>>>> kernel`_sx_xunlock 7 0.0% >>>>>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>>>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>>>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>>>>> vmm.ko`lapic_timer_tick 8 0.0% >>>>>>> zfs.ko`list_next 8 0.0% >>>>>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>>>>> kernel`userret 9 0.0% >>>>>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>>>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>>>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>>>>> vmm.ko`vm_lapic 11 0.0% >>>>>>> kernel`spinlock_enter 14 0.0% >>>>>>> kernel`__mtx_lock_sleep 15 0.0% >>>>>>> vmm.ko`vcpu_set_state 16 0.0% >>>>>>> vmm.ko`vm_nmi_pending 16 0.0% >>>>>>> kernel`malloc_type_zone_******allocated 16 >>>>>>> 0.0% >>>>>>> >>>>>>> kernel`devfs_ioctl_f 16 0.0% >>>>>>> kernel`pmap_kextract 17 0.0% >>>>>>> kernel`sys_ioctl 18 0.0% >>>>>>> kernel`malloc 21 0.0% >>>>>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>>>>> kernel`uma_zfree_arg 28 0.0% >>>>>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>>>>> kernel`trap 30 0.0% >>>>>>> kernel`hpet_get_timecount 31 0.0% >>>>>>> dtrace.ko`dtrace_trap 35 0.0% >>>>>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>>>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>>>>> kernel`copyout 43 0.0% >>>>>>> kernel`critical_enter 44 0.0% >>>>>>> kernel`copyin 51 0.1% >>>>>>> vmm.ko`vmcs_read 53 0.1% >>>>>>> kernel`fpusave 61 0.1% >>>>>>> kernel`uma_zalloc_arg 62 0.1% >>>>>>> kernel`devvn_refthread 69 0.1% >>>>>>> kernel`atomic_load_acq_int 70 0.1% >>>>>>> kernel`fpurestore 70 0.1% >>>>>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>>>>> kernel`amd64_syscall 77 0.1% >>>>>>> kernel`bcopy 91 0.1% >>>>>>> kernel`fpudna 103 0.1% >>>>>>> kernel`free 103 0.1% >>>>>>> kernel`critical_exit 105 0.1% >>>>>>> kernel`dev_relthread 108 0.1% >>>>>>> kernel`spinlock_exit 119 0.1% >>>>>>> kernel`kern_ioctl 138 0.1% >>>>>>> kernel`fpuexit 242 0.3% >>>>>>> kernel`0xffffffff80 306 0.3% >>>>>>> vmm.ko`restore_host_msrs 398 0.4% >>>>>>> vmm.ko`vm_run 428 0.4% >>>>>>> vmm.ko`restore_guest_msrs 433 0.5% >>>>>>> kernel`cpu_idle 906 1.0% >>>>>>> vmm.ko`vmx_run 9266 9.7% >>>>>>> kernel`acpi_cpu_c1 81134 85.2% >>>>>>> >>>>>>> Hope this helps, let me know if you have any ideas on what to try >>>>>>> next. >>>>>>> >>>>>>> >>>>>>> Could you try the following command to get vcpu stats? >>>>>>> >>>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>>>>> >>>>>> best >>>>>> Neel >>>>>> >>>>>> >>>>>> Dan >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>>>>> >>>>>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>>>>> >>>>>>> >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> >>>>>>>>> Any ideas? >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Just to confirm: your host system is an i7 3930k, which looks >>>>>>>>> like a 6 >>>>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>>>>> >>>>>>>>> >>>>>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>>>>> >>>>>>>> The issue you are seeing looks a bit like what happens when >>>>>>>> systems are >>>>>>>> >>>>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>>>>> >>>>>>>>> >>>>>>>>> I was running in on an idle system; only bhyve was running. >>>>>>>> >>>>>>>> >>>>>>>> How much spare RAM is on the host when you started the vm ? Also, >>>>>>>> is >>>>>>>> >>>>>>>> there much running on the host when the guest is started ? >>>>>>>>> >>>>>>>>> >>>>>>>>> 64GB >>>>>>>> >>>>>>>> Here's some more information (sorry for the formatting): >>>>>>>> >>>>>>>> Case 1: single vCPU via: >>>>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio >>>>>>>> coco >>>>>>>> >>>>>>>> Top shows it idle (note low load, 4GB memory given to guest, still >>>>>>>> 53GB >>>>>>>> free RAM, nothing else clocking CPU cycles): >>>>>>>> >>>>>>>> root@olive:~ # top -bH >>>>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>>>>> 14:11:51 >>>>>>>> 24 processes: 1 running, 23 sleeping >>>>>>>> >>>>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M >>>>>>>> Other >>>>>>>> Swap: >>>>>>>> >>>>>>>> >>>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>>> COMMAND >>>>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>>>>> bhyve{coco vcpu 0} >>>>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% >>>>>>>> syslogd >>>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>>>>> bhyve{coco mevent} >>>>>>>> >>>>>>>> >>>>>>>> So, case #1 all is normal. I was able to build a kernel with very >>>>>>>> good >>>>>>>> performance. >>>>>>>> >>>>>>>> The boot sequence looks like the following: >>>>>>>> >>>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>>> Launching virtual machine "coco" ... >>>>>>>> Consoles: userboot >>>>>>>> >>>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>>> >>>>>>>> / >>>>>>>> ______ ____ _____ _____ >>>>>>>> | ____| | _ \ / ____| __ \ >>>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>>> | | | | | | || | | | >>>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>>> ` >>>>>>>> s` >>>>>>>> `.....---.......--.``` >>>>>>>> -/ >>>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>>> +. >>>>>>>> | | yo`:. :o >>>>>>>> `+- >>>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>>> -o/ >>>>>>>> | 2. Boot [S]ingle User | .- >>>>>>>> ::/sy+:. >>>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>>> `-- >>>>>>>> / >>>>>>>> | 4. Reboot | `: >>>>>>>> :` >>>>>>>> | | `: >>>>>>>> :` >>>>>>>> | Options: | / >>>>>>>> / >>>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>>> -. >>>>>>>> | | -- -. >>>>>>>> | | `:` >>>>>>>> `:` >>>>>>>> | | .-- >>>>>>>> `--. >>>>>>>> | | .---.....----. >>>>>>>> +-----------------------------******------------+ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Booting... >>>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>>> 1994 >>>>>>>> The Regents of the University of California. All rights >>>>>>>> reserved. >>>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>>> >>>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class >>>>>>>> CPU) >>>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>>> Stepping = 7 >>>>>>>> Features=0x8fa3ab7f<FPU,VME,******DE,PSE,TSC,MSR,PAE,CX8,APIC,**** >>>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>>> Features2=0x83bee217<SSE3,******PCLMULQDQ,DTES64,DS_CPL,SSSE3,** >>>>>>>> **** >>>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>>> TSCDLT,**AESNI,HV> >>>>>>>> AMD Features=0x2c100800<SYSCALL,******NX,Page1GB,RDTSCP,LM> >>>>>>>> >>>>>>>> AMD Features2=0x1<LAHF> >>>>>>>> TSC: P-state invariant >>>>>>>> real memory = 5368709120 (5120 MB) >>>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>>> Event timer "LAPIC" quality 400 >>>>>>>> ACPI APIC Table: <BHYVE BVMADT > >>>>>>>> random device not loaded; using insecure entropy >>>>>>>> ioapic0: Changing APIC ID to 1 >>>>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard >>>>>>>> kbd0 at kbdmux0 >>>>>>>> acpi0: <BHYVE BVXSDT> on motherboard >>>>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 >>>>>>>> pci0: <ACPI PCI bus> on pcib0 >>>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem >>>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0 >>>>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress> >>>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>>> <Status,MrgRxBuf,MacAddress> >>>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem >>>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1 >>>>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs> >>>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>>> <RingIndirect,MaxNumSegs> >>>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 >>>>>>>> on >>>>>>>> pci0 >>>>>>>> uart2: console (9600,n,8,1) >>>>>>>> Timecounters tick every 10.000 msec >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>>> >>>>>>>> Setting hostid: 0x24bfd025. >>>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>>> Starting file system checks: >>>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>>>>> fragmentation) >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> Mounting local file systems:. >>>>>>>> Writing entropy file:. >>>>>>>> Setting hostname: cocopuff.example.com. >>>>>>>> vtnet0: link state changed to UP >>>>>>>> Starting Network: lo0 vtnet0. >>>>>>>> lo0: flags=8049<UP,LOOPBACK,******RUNNING,MULTICAST> metric 0 mtu >>>>>>>> 16384 >>>>>>>> options=600003<RXCSUM,TXCSUM,******RXCSUM_IPV6,TXCSUM_IPV6> >>>>>>>> >>>>>>>> inet6 ::1 prefixlen 128 >>>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>>> nd6 options=21<PERFORMNUD,AUTO_******LINKLOCAL> >>>>>>>> vtnet0: flags=8943<UP,BROADCAST,******RUNNING,PROMISC,SIMPLEX,**** >>>>>>>> MULTICAST> >>>>>>>> metric 0 mtu 1500 >>>>>>>> options=80028<VLAN_MTU,JUMBO_******MTU,LINKSTATE> >>>>>>>> >>>>>>>> ether 00:a0:98:ac:0d:df >>>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>>> 172.22.255.255 >>>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid >>>>>>>> 0x1 >>>>>>>> nd6 options=29<PERFORMNUD,******IFDISABLED,AUTO_LINKLOCAL> >>>>>>>> >>>>>>>> media: Ethernet 1000baseT <full-duplex> >>>>>>>> status: active >>>>>>>> Starting devd. >>>>>>>> add net default: gateway 172.22.22.252 >>>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>>> add net fe80::: gateway ::1 >>>>>>>> add net ff02::: gateway ::1 >>>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>>> Creating and/or trimming log files. >>>>>>>> Starting syslogd. >>>>>>>> Clearing /tmp (X related). >>>>>>>> Updating motd:. >>>>>>>> Performing sanity check on sshd configuration. >>>>>>>> Starting sshd. >>>>>>>> Starting cron. >>>>>>>> Starting background file system checks in 60 seconds. >>>>>>>> >>>>>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>>>>> >>>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>>> >>>>>>>> login: >>>>>>>> >>>>>>>> >>>>>>>> ====== Next, boot the same image with two (2) vCPU: >>>>>>>> >>>>>>>> Command run by vmrun.sh: >>>>>>>> >>>>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio >>>>>>>> coco >>>>>>>> >>>>>>>> Top shows it running out of control: >>>>>>>> >>>>>>>> root@olive:~ # top -bH >>>>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>>>>> 14:17:32 >>>>>>>> 25 processes: 3 running, 22 sleeping >>>>>>>> >>>>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M >>>>>>>> Other >>>>>>>> Swap: >>>>>>>> >>>>>>>> >>>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>>> COMMAND >>>>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>>>>> bhyve{coco vcpu 1} >>>>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>>>>> bhyve{coco vcpu 0} >>>>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% >>>>>>>> syslogd >>>>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>>>>> bhyve{coco mevent} >>>>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>>>>> >>>>>>>> Even though, it's just sitting at the login prompt: >>>>>>>> >>>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>>> Launching virtual machine "coco" ... >>>>>>>> Consoles: userboot >>>>>>>> >>>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>>> >>>>>>>> / >>>>>>>> ______ ____ _____ _____ >>>>>>>> | ____| | _ \ / ____| __ \ >>>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>>> | | | | | | || | | | >>>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>>> ` >>>>>>>> s` >>>>>>>> `.....---.......--.``` >>>>>>>> -/ >>>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>>> +. >>>>>>>> | | yo`:. :o >>>>>>>> `+- >>>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>>> -o/ >>>>>>>> | 2. Boot [S]ingle User | .- >>>>>>>> ::/sy+:. >>>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>>> `-- >>>>>>>> / >>>>>>>> | 4. Reboot | `: >>>>>>>> :` >>>>>>>> | | `: >>>>>>>> :` >>>>>>>> | Options: | / >>>>>>>> / >>>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>>> -. >>>>>>>> | | -- -. >>>>>>>> | | `:` >>>>>>>> `:` >>>>>>>> | | .-- >>>>>>>> `--. >>>>>>>> | | .---.....----. >>>>>>>> +-----------------------------******------------+ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Booting... >>>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>>> 1994 >>>>>>>> The Regents of the University of California. All rights >>>>>>>> reserved. >>>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>>> >>>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class >>>>>>>> CPU) >>>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>>> Stepping = 7 >>>>>>>> Features=0x8fa3ab7f<FPU,VME,******DE,PSE,TSC,MSR,PAE,CX8,APIC,**** >>>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>>> Features2=0x83bee217<SSE3,******PCLMULQDQ,DTES64,DS_CPL,SSSE3,** >>>>>>>> **** >>>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>>> TSCDLT,**AESNI,HV> >>>>>>>> AMD Features=0x2c100800<SYSCALL,******NX,Page1GB,RDTSCP,LM> >>>>>>>> >>>>>>>> AMD Features2=0x1<LAHF> >>>>>>>> TSC: P-state invariant >>>>>>>> real memory = 5368709120 (5120 MB) >>>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>>> Event timer "LAPIC" quality 400 >>>>>>>> ACPI APIC Table: <BHYVE BVMADT > >>>>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>>>>> cpu0 (BSP): APIC ID: 0 >>>>>>>> cpu1 (AP): APIC ID: 1 >>>>>>>> random device not loaded; using insecure entropy >>>>>>>> ioapic0: Changing APIC ID to 2 >>>>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard >>>>>>>> kbd0 at kbdmux0 >>>>>>>> acpi0: <BHYVE BVXSDT> on motherboard >>>>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 >>>>>>>> pci0: <ACPI PCI bus> on pcib0 >>>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem >>>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0 >>>>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress> >>>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>>> <Status,MrgRxBuf,MacAddress> >>>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem >>>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1 >>>>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs> >>>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>>> <RingIndirect,MaxNumSegs> >>>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 >>>>>>>> on >>>>>>>> pci0 >>>>>>>> uart2: console (9600,n,8,1) >>>>>>>> Timecounters tick every 10.000 msec >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> SMP: AP CPU #1 Launched! >>>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>>> >>>>>>>> Setting hostid: 0x24bfd025. >>>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>>> Starting file system checks: >>>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>>>>> fragmentation) >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>>> Mounting local file systems:. >>>>>>>> Writing entropy file:. >>>>>>>> Setting hostname: cocopuff.example.com. >>>>>>>> vtnet0: link state changed to UP >>>>>>>> Starting Network: lo0 vtnet0. >>>>>>>> lo0: flags=8049<UP,LOOPBACK,******RUNNING,MULTICAST> metric 0 mtu >>>>>>>> 16384 >>>>>>>> options=600003<RXCSUM,TXCSUM,******RXCSUM_IPV6,TXCSUM_IPV6> >>>>>>>> >>>>>>>> inet6 ::1 prefixlen 128 >>>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>>> nd6 options=21<PERFORMNUD,AUTO_******LINKLOCAL> >>>>>>>> vtnet0: flags=8943<UP,BROADCAST,******RUNNING,PROMISC,SIMPLEX,**** >>>>>>>> MULTICAST> >>>>>>>> metric 0 mtu 1500 >>>>>>>> options=80028<VLAN_MTU,JUMBO_******MTU,LINKSTATE> >>>>>>>> >>>>>>>> ether 00:a0:98:ac:0d:df >>>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>>> 172.22.255.255 >>>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid >>>>>>>> 0x1 >>>>>>>> nd6 options=29<PERFORMNUD,******IFDISABLED,AUTO_LINKLOCAL> >>>>>>>> >>>>>>>> media: Ethernet 1000baseT <full-duplex> >>>>>>>> status: active >>>>>>>> Starting devd. >>>>>>>> add net default: gateway 172.22.22.252 >>>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>>> add net fe80::: gateway ::1 >>>>>>>> add net ff02::: gateway ::1 >>>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>>> Creating and/or trimming log files. >>>>>>>> Starting syslogd. >>>>>>>> syslogd: timed out waiting for child >>>>>>>> /etc/rc: WARNING: failed to start syslogd >>>>>>>> Clearing /tmp (X related). >>>>>>>> Updating motd:. >>>>>>>> Performing sanity check on sshd configuration. >>>>>>>> Starting sshd. >>>>>>>> Starting cron. >>>>>>>> Starting background file system checks in 60 seconds. >>>>>>>> >>>>>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>>>>> >>>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>>> >>>>>>>> login: >>>>>>>> >>>>>>>> >>>>>>>> ========== end of boot sequence >>>>>>>> >>>>>>>> So, something is broken with SMP on the guest, I'd be happy to help >>>>>>>> just >>>>>>>> let me know. >>>>>>>> >>>>>>>> Dan >>>>>>>> >>>>>>>> ______________________________******_________________ >>>>>>>> freebsd-virtualization@****freeb**sd.org <http://freebsd.org>; < >>>>>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org><; >>>>>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization@freebsd.org> >>>>>>>> >>mailing >>>>>>>> list >>>>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-****<http://lists.freebsd.org/****mailman/listinfo/freebsd-****>; >>>>>>>> virtualization<http://lists.**freebsd.org/**mailman/** >>>>>>>> listinfo/freebsd-****virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>; >>>>>>>> > >>>>>>>> <http://lists.**freebsd.org/**mailman/listinfo/**freebsd-** >>>>>>>> virtualization<http://freebsd.org/mailman/listinfo/**freebsd-virtualization>; >>>>>>>> <http://lists.**freebsd.org/mailman/listinfo/** >>>>>>>> freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>; >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>>>> unsubscribe@freebsd.org <freebsd-virtualization-** >>>>>>>> unsubscribe@freebsd.org<freebs**d-virtualization-unsubscribe@** >>>>>>>> freebsd.org <freebsd-virtualization-unsubscribe@freebsd.org>> >>>>>>>> >>>>>>>>> " >>>>>>>>> >>>>>>>> >>>>>>>> ______________________________******_________________ >>>>>>>> >>>>>>>> freebsd-virtualization@****freeb**sd.org <http://freebsd.org>; < >>>>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org><; >>>>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization@freebsd.org> >>>>>>> >>mailing >>>>>>> list >>>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-****** >>>>>>> virtualization<http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization>; >>>>>>> <http://lists.**freebsd.org/**mailman/**listinfo/freebsd-**** >>>>>>> virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>; >>>>>>> > >>>>>>> <http://lists.**freebsd.org/**mailman/listinfo/**freebsd-** >>>>>>> virtualization<http://freebsd.org/mailman/listinfo/**freebsd-virtualization>; >>>>>>> <http://lists.**freebsd.org/mailman/listinfo/** >>>>>>> freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>; >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>>> unsubscribe@freebsd.org <freebsd-virtualization-** >>>>>>> unsubscribe@freebsd.org <freebsd-virtualization-** >>>>>>> unsubscribe@freebsd.org<freebsd-virtualization-unsubscribe@freebsd.org> >>>>>>> > >>>>>>> >>>>>>>> " >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> ______________________________**_________________ >>> freebsd-virtualization@**freebsd.org<freebsd-virtualization@freebsd.org>mailing list >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>; >>> To unsubscribe, send any mail to "freebsd-virtualization-** >>> unsubscribe@freebsd.org <freebsd-virtualization-unsubscribe@freebsd.org> >>> " >>> >>> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFgRE9Hv6aHbqp0c7BS=siO%2Br8sQGBPSkHYUAi%2B6_MLZHb97gQ>