From owner-freebsd-virtualization@freebsd.org Tue Oct 13 21:15:45 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 628FAA124D8 for ; Tue, 13 Oct 2015 21:15:45 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: from mail-pa0-x236.google.com (mail-pa0-x236.google.com [IPv6:2607:f8b0:400e:c03::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 345501A1B for ; Tue, 13 Oct 2015 21:15:45 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: by padcn9 with SMTP id cn9so916870pad.2 for ; Tue, 13 Oct 2015 14:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pluribusnetworks.com; s=google; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=QteF8JTn5xwYp7vqJ7FOtZUD0LnGSf9kGzS04fJ1y70=; b=JqPYlskiggYhQjuE71Bi7VfCrDHE+R+FdqcM1JSaox8E7wgNi82efbDoiqB9jdsW8g eyOu29l4fqZ/rGoBdPagK5U6IU9rU7XaiHUaAPGe5ALV3fA14Q3aOSEWsctdOENKRlmE dnCMs2LBd/oQuZgp9/IH5DecxC0b4pRbr6AnA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=QteF8JTn5xwYp7vqJ7FOtZUD0LnGSf9kGzS04fJ1y70=; b=MohxMmAdtloKWdWavmG5awTC7VTjaOMaeMN8JR7AC9uHvyYs4YufNkqYYLEuxH10TN yqM9FTP9Kp12VP3VFR3aXeOUDrywvIWhOnoTWwk9UvLG8VAVHmZGk06njPZxxvGtVUB6 GGt6DSms+KwPDIcGXUkg9yj6nltu3H6lwWs2sMDvmcp5AMdERbdjInh3YvOOzX7/quXX B+7VhEEP6hKJn/H9NXgar6OYbg9jLd+Pq4vuum3qPzaU9/utW54+SBheBPGHXEDFft8F Aqz9CfjOFYehDn/AfhHUOkytivZTyuPem/d3IoU6nyY3icK4QcXh3fbPRR9TEAbNPS4V 7M4g== X-Gm-Message-State: ALoCoQl3UdKDtN/OJOBwdkcmLbezY8l7PYNXAmC7HmpHK1zpGKmXCg5LHsNy/cAoue5gT2Bomgbe X-Received: by 10.68.242.2 with SMTP id wm2mr43340551pbc.31.1444770944357; Tue, 13 Oct 2015 14:15:44 -0700 (PDT) Received: from [10.0.1.16] (209-6-121-211.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com. [209.6.121.211]) by smtp.gmail.com with ESMTPSA id fb1sm5613020pab.9.2015.10.13.14.15.43 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 13 Oct 2015 14:15:43 -0700 (PDT) Subject: Re: Illumos boot Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Tycho Nightingale In-Reply-To: <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> Date: Tue, 13 Oct 2015 17:15:43 -0400 Cc: "freebsd-virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> To: Matt Churchyard X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 21:15:45 -0000 Hi, On Oct 13, 2015, at 9:35 AM, Matt Churchyard wrote: >=20 >> On Oct 13, 2015, at 7:17 AM, Matt Churchyard via = freebsd-virtualization wrote: >=20 >> In my quest to continue expanding guest support in my vm-bhyve = utility (See https://github.com/churchers/vm-bhyve :) ), I've found the = Windows support pretty solid once I got clear on the slot requirements. = I'm now trying an OS that requires CSM (Illumos) but unfortunately I'm = currently struggling to get it to boot up correctly. >>=20 >> Here's an example of the command I'm generating at the moment (This = is running on an Intel Core-i3): >>=20 >> bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ >> -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ >> -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ >> -s 5:0,virtio-net,tap0 \ >> -l com1,stdio -l com2,/dev/nmdm2A \ >> -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ >> smartos >>=20 >> I have com1 set to stdio so I can easily watch the output as it runs. >> It tends to get as far as "Legacy INT19 Boot...", then fall over. >> Depending on whether I put the network interface directly in the slot = after the HDD, I seem to get different errors - >>=20 >> slot 3 - cd >> slot 4 - hdd >> slot 5 - virtio-net >>=20 >> panic[cpu0]/thread=3Dffffff01457cdb40: BAD TRAP: type=3De (#pf Page = fault) rp=3Dffffff0004a69a60 addr=3D40 occurred in module "genunix" due = to a NULL pointer dereference >>=20 >> slot 3 - cd >> slot 4 - hdd >> slot 7 - virtio-net >>=20 >> panic[cpu1]/thread=3Dffffff0004002c40: BAD TRAP: type=3Dd (#gp = General protection) rp=3Dffffff0004002740 addr=3D0 >>=20 >> On com2 I see the boot menu, then one and a half lines of dots. The = second line of dots stops about 2/3 of the way across. >=20 >> Have you tried booting illumos in verbose mode - edit the grub = command line and provide '-v'. That may give you a better backtrace = than a >program counter. >=20 > This is what I get from a verbose boot: >=20 > Bhyve-HandleProtocol: Copying DevPath: = PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0x0,0x0) [32] > Legacy INT19 Boot... > cpu0: x86 (chipid 0x0 GenuineIntel 206A7 family 6 model 42 step 7 = clock 3109 MHz) > cpu0: Intel(r) Core(tm) i3-2100 CPU @ 3.10GHz > pseudo-device: stmf_sbd0 > stmf_sbd0 is /pseudo/stmf_sbd@0 > pseudo-device: lofi0 > lofi0 is /pseudo/lofi@0 > pseudo-device: devinfo0 > devinfo0 is /pseudo/devinfo@0 > acpinex0 at root > acpinex0 is /fw > iscsi0 at root > iscsi0 is /iscsi > xsvc0 at root: space 0 offset 0 > xsvc0 is /xsvc@0,0 > acpinex: sb@0, acpinex1 > acpinex1 is /fw/sb@0 > pseudo-device: pseudo1 > pseudo1 is /pseudo/zconsnex@1 > pseudo-device: pseudo2 > pseudo2 is /pseudo/zfdnex@2 > /pci@0,0/pci8086,2821@3 : > SATA CD/DVD (ATAPI) device at port 0 > model BHYVE SATA DVD ROM > firmware 001 > serial number BHYVE-EA14-A68A-54FA > supported features: > DMA > SATA Gen3 signaling speed (6.0Gbps) > pseudo-device: llc10 > llc10 is /pseudo/llc1@0 > pseudo-device: power0 > power0 is /pseudo/power@0 > pseudo-device: ramdisk1024 > ramdisk1024 is /pseudo/ramdisk@1024 > pseudo-device: ucode0 > ucode0 is /pseudo/ucode@0 > pseudo-device: zfs0 > zfs0 is /pseudo/zfs@0 > pseudo-device: srn0 > srn0 is /pseudo/srn@0 > pseudo-device: dtrace0 > dtrace0 is /pseudo/dtrace@0 > pseudo-device: dcpc0 > dcpc0 is /pseudo/dcpc@0 > pseudo-device: fasttrap0 > fasttrap0 is /pseudo/fasttrap@0 > pseudo-device: fbt0 > fbt0 is /pseudo/fbt@0 > pseudo-device: profile0 > profile0 is /pseudo/profile@0 > pseudo-device: lockstat0 > lockstat0 is /pseudo/lockstat@0 > pseudo-device: sdt0 > sdt0 is /pseudo/sdt@0 > pseudo-device: systrace0 > systrace0 is /pseudo/systrace@0 > pseudo-device: ipd0 > ipd0 is /pseudo/ipd@0 > pseudo-device: stmf0 > stmf0 is /pseudo/stmf@0 > sd0 at ahci0: target 0 lun 0 > sd0 is /pci@0,0/pci8086,2821@3/cdrom@0,0 > pseudo-device: fssnap0 > fssnap0 is /pseudo/fssnap@0 > /pci@0,0/pci8086,2821@3/cdrom@0,0 (sd0) online > /pci@0,0/pci8086,2821@4 : > SATA disk device at port 0 > model BHYVE SATA DISK > firmware 001 > serial number BHYVE-3083-1AF1-1754 > supported features: > 48-bit LBA, DMA, Native Command Queueing > SATA Gen3 signaling speed (6.0Gbps) > Supported queue depth 32 > capacity =3D 62914560 sectors > WARNING: kvm: no hardware support >=20 > pseudo-device: pool0 > pool0 is /pseudo/pool@0 > pseudo-device: bpf0 > bpf0 is /pseudo/bpf@0 > sd1 at ahci1: target 0 lun 0 > sd1 is /pci@0,0/pci8086,2821@4/disk@0,0 > pseudo-device: pm0 > pm0 is /pseudo/pm@0 > pseudo-device: nsmb0 > nsmb0 is /pseudo/nsmb@0 > pseudo-device: tap0 > tap0 is /pseudo/tap@0 > /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online > NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff > pseudo-device: tun0 > tun0 is /pseudo/tun@0 > pseudo-device: lx_systrace0 > lx_systrace0 is /pseudo/lx_systrace@0 >=20 > panic[cpu0]/thread=3Dffffff0002566c40: BAD TRAP: type=3Dd (#gp General = protection) rp=3Dffffff00025664c0 addr=3D20 >=20 > sched: #gp General protection > addr=3D0x20 > pid=3D0, pc=3D0xfffffffff80d375a, sp=3D0xffffff00025665b0, = eflags=3D0x10282 > cr0: 8005003b cr4: = 406b8 > cr2: fed3b5accr3: 1dc00000cr8: c >=20 > rdi: 7f1a90ffffff00c3 rsi: ffffff00c33321f8 rdx: = ffffff00c37f5828 > rcx: ffffff00c3868603 r8: ffffff00ca7aa600 r9: = 2ba6 > rax: 0 rbx: ffffff00c30d0ef0 rbp: = ffffff00025665c0 > r10: fffffffffb8554c4 r11: 1 r12: = 1f > r13: ffffff00c319f880 r14: 10 r15: = 20 > fsb: 0 gsb: fffffffffbc326a0 ds: = 4b > es: 4b fs: 0 gs: = 1c3 > trp: d err: 0 rip: = fffffffff80d375a > cs: 30 rfl: 10282 rsp: = ffffff00025665b0 > ss: 38 >=20 > This is the log of the bhyve options used (apart from 1 cpu, 1G ram) >=20 > Oct 13 14:22:43: [bhyve devices: -s 0,hostbridge -s 31,lpc -s = 4:0,ahci-hd,/data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] > Oct 13 14:22:43: [bhyve console: -l com1,/dev/nmdm1A -l = com2,/dev/nmdm2A] > Oct 13 14:22:43: [bhyve options: -Hw -l = bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd] > Oct 13 14:22:43: [bhyve iso device: -s = 3:0,ahci-cd,/data/vm/.iso/smartos-latest.iso] Ouch, even with the additional verbosity, the output isn't very = insightful. All I can glean is that you are reasonably far along in the = boot process. You could try to run with KMDB (-k boot option) and do = some disassembly around that program counter to see if any specific = module is implicated. Alternately, I'd omit the network device and see how far that gets you. >> Interestingly, my code normally puts the CD after the HDD, which = Windows seems happy with as long as the slots are consecutive. >> In SmartOS this gives me a different error: >>=20 >> slot 3 - hdd >> slot 4 - cd >> slot 5 - virtio-net >>=20 >> PlatformBdsBootFail >> Boot Failed. Harddisk 1 >> !!!! Find PE image = /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC48/X64/UefiCpuPkg/Cpu= Dxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=3D000000007F8DC000, = EntryPoint=3D000000007F8DC2AF) !!!! >=20 >> This error is from the UEFI code. It implies that the CSM boot = failed or was never invoked. If the HDD isn't bootable, yet the CD is, = that is the >most likely source as the CSM assumes the first block = device it encounters is the desired boot source. >=20 > Ok, so the boot semantics are currently different between the CSM and = non-CSM firmware? CSM will try and boot the first device and fail if = it's not bootable, whereas non-CSM will always boot CD if it's bootable, = regardless of order (from Windows instructions). Yes, the boot semantics are different. The UEFI (non-CSM) path is = somewhat more tolerant of trying the next device in the boot-order if = the current one is deemed "unbootable". The CSM path is cruder in that = it just searches for the first block device and goes for it. If that = device is unbootable it will fall thorough to the UEFI path somewhat = ungracefully as UEFI isn't really expecting CSM to ever return. Tycho=