From owner-freebsd-xen@freebsd.org Mon Jan 23 15:26:04 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A550CBE789 for ; Mon, 23 Jan 2017 15:26:04 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 16FCA1E25 for ; Mon, 23 Jan 2017 15:26:03 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1485185158; s=zoho; d=nfvexpress.com; i=alexander.nusov@nfvexpress.com; h=From:Content-Type:Mime-Version:Subject:Message-Id:Date:To; l=72291; bh=oB6j4RNPkqjoY5uEcitCXCdgQonlYqHOmOv5Nh9wqI4=; b=oRy4Or7DfIOpqhKCmwGHJ4CF9oF1oUyKXqKkyoQC5q2JmDHTm31N72IzkLHs099J qanU6Y0hxqFNB28gOEI+sJb7GuWqszZ9B4udJjfE+W8B4/Qt56Kh0VBOWM0ODVtqH5X 6ZqQDuY1ZC4FXRYVJv+4r16puAZ31iStimhplAvs= Received: from [192.168.1.66] (95-30-222-83.broadband.corbina.ru [95.30.222.83]) by mx.zohomail.com with SMTPS id 1485185158844965.1284840212834; Mon, 23 Jan 2017 07:25:58 -0800 (PST) From: Alexander Nusov Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) Message-Id: Date: Mon, 23 Jan 2017 18:25:23 +0300 To: freebsd-xen@freebsd.org X-Mailer: Apple Mail (2.3259) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jan 2017 15:26:04 -0000 Hello, Sorry for cross-posting, since it's related to Xen hypervisor I'm = forwarding this message to the freebsd-xen mailing list. I'm trying to launch a HVM DomU guest from QCOW2 image by using PV = driver on FreeBSD 11 Dom0. The issue is that guest cannot connect to the device/vbd and requires to = wait for 4 minutes to proceed, it goes through the countdown and starts = fine (disk, networking) [ 6.684115] XENBUS: Waiting for devices to initialise: = 25s...20s...15s...10s...5s...0s...235s...230s...225s...220s...215s...210s.= ..205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...15= 5s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s..= .100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...= 40s...35s...30s...25s...20s...15s...10s...5s...0s... [ 271.591403] XENBUS: Timeout connecting to device: device/vbd/51712 = (local state 3, remote state 1) [ 271.599963] XENBUS: Device with no driver: device/vkbd/0 [ 271.604249] Magic number: 1:453:334 ... login:=20 Unlike Linux It's impossible to boot FreeBSD 11 guests from QCOW2 = (xenbusb_nop_confighook_cb timeout) Steps to reproduce: 1. Download qcow2 cirros image (small linux)=20 http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img = = > # file cirros-0.3.4-x86_64-disk.img=20 cirros-0.3.4-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes 2. create DomU from config bellow xl create -c config.cfg builder =3D "hvm" memory =3D 512 vcpus =3D 2 name =3D "cirros" disk =3D [ 'file:qcow2:/root/cirros-0.3.4-x86_64-disk.img,xvda,w' ] boot =3D "c"=20 vnc =3D 1 vnclisten =3D "0.0.0.0" usbdevice =3D 'tablet' on_poweroff =3D 'destroy' on_reboot =3D 'restart' on_crash =3D 'restart' acpi =3D 1 serial =3D 'pty' I've also tried multiple configurations like tap:qcow2:. tap2:qcow2:, = aio:, switching from xen bus to ide. didn't work. The only driver that had no issues was PHY but it supports only RAW = images. Is that a bug or I'm missing something? tested both STABLE snapshot and 11.0-RELEASE # uname -a FreeBSD xen 11.0-STABLE FreeBSD 11.0-STABLE #0 r311441: Thu Jan 5 = 22:45:20 UTC 2017 # pkg info | grep xen xen-4.7.0_2 Xen Hypervisor meta port xen-kernel-4.7.1_3 Hypervisor using a microkernel design xen-tools-4.7.1_1 Xen management tool, based on LibXenlight # cat /boot/loader.conf=20 hw.pci.mcfg=3D0 xen_kernel=3D"/boot/xen" xen_cmdline=3D"dom0_mem=3D8192M dom0_max_vcpus=3D8 dom0pvh=3D1 = com1=3D115200,8n1 guest_loglvl=3Dall loglvl=3Dall" # xenstore-ls -p 4 =3D "" . . . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) vm =3D "/vm/94b147df-e15d-11e6-a685-002590d7eca6" . . . . . (n0,r4) name =3D "cirros" . . . . . . . . . . . . . . . . . . . . . (n0,r4) cpu =3D "" . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) 0 =3D "" . . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) availability =3D "online" . . . . . . . . . . . . . . . . (n0,r4) 1 =3D "" . . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) availability =3D "online" . . . . . . . . . . . . . . . . (n0,r4) memory =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) static-max =3D "4194304" . . . . . . . . . . . . . . . . . (n0,r4) target =3D "4186112" . . . . . . . . . . . . . . . . . . . (n0,r4) videoram =3D "8192" . . . . . . . . . . . . . . . . . . . (n0,r4) device =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) suspend =3D "" . . . . . . . . . . . . . . . . . . . . . . (n0,r4) event-channel =3D "" . . . . . . . . . . . . . . . . . . (n4) vbd =3D "" . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) 51712 =3D "" . . . . . . . . . . . . . . . . . . . . . . (n4,r0) backend =3D "/local/domain/0/backend/qdisk/4/51712" . . (n4,r0) backend-id =3D "0" . . . . . . . . . . . . . . . . . . . (n4,r0) state =3D "3" . . . . . . . . . . . . . . . . . . . . . (n4,r0) virtual-device =3D "51712" . . . . . . . . . . . . . . . (n4,r0) device-type =3D "disk" . . . . . . . . . . . . . . . . . (n4,r0) ring-ref =3D "8" . . . . . . . . . . . . . . . . . . . . (n4,r0) event-channel =3D "18" . . . . . . . . . . . . . . . . . (n4,r0) protocol =3D "x86_64-abi" . . . . . . . . . . . . . . . (n4,r0) vkbd =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) 0 =3D "" . . . . . . . . . . . . . . . . . . . . . . . . (n4,r0) backend =3D "/local/domain/0/backend/vkbd/4/0" . . . . . (n4,r0) backend-id =3D "0" . . . . . . . . . . . . . . . . . . . (n4,r0) state =3D "1" . . . . . . . . . . . . . . . . . . . . . (n4,r0) control =3D "" . . . . . . . . . . . . . . . . . . . . . . (n0,r4) shutdown =3D "" . . . . . . . . . . . . . . . . . . . . . (n4) platform-feature-multiprocessor-suspend =3D "1" . . . . . (n0,r4) platform-feature-xs_reset_watches =3D "1" . . . . . . . . (n0,r4) hvmloader =3D "" . . . . . . . . . . . . . . . . . . . . . (n0,r4) bios =3D "seabios" . . . . . . . . . . . . . . . . . . . . (n0,r4) allow-memory-relocate =3D "0" . . . . . . . . . . . . . . (n0,r4) data =3D "" . . . . . . . . . . . . . . . . . . . . . . . . (n4) drivers =3D "" . . . . . . . . . . . . . . . . . . . . . . (n4) feature =3D "" . . . . . . . . . . . . . . . . . . . . . . (n4) attr =3D "" . . . . . . . . . . . . . . . . . . . . . . . . (n4) domid =3D "4" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) store =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) port =3D "1" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) ring-ref =3D "1044476" . . . . . . . . . . . . . . . . . . (n0,r4) platform =3D "" . . . . . . . . . . . . . . . . . . . . . . (n0,r4) acpi =3D "1" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) acpi_s3 =3D "1" . . . . . . . . . . . . . . . . . . . . . (n0,r4) acpi_s4 =3D "1" . . . . . . . . . . . . . . . . . . . . . (n0,r4) console =3D "" . . . . . . . . . . . . . . . . . . . . . . (n0,r4) backend =3D "/local/domain/0/backend/console/4/0" . . . . (n0,r4) backend-id =3D "0" . . . . . . . . . . . . . . . . . . . . (n4,r0) limit =3D "1048576" . . . . . . . . . . . . . . . . . . . (n0,r4) type =3D "xenconsoled" . . . . . . . . . . . . . . . . . . (n0,r4) output =3D "pty" . . . . . . . . . . . . . . . . . . . . . (n0,r4) tty =3D "/dev/pts/1" . . . . . . . . . . . . . . . . . . . (n0,r4) port =3D "2" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) ring-ref =3D "1044479" . . . . . . . . . . . . . . . . . . (n0,r4) vnc-listen =3D "0.0.0.0" . . . . . . . . . . . . . . . . . (n0,r4) vnc-port =3D "5900" . . . . . . . . . . . . . . . . . . . (n0,r4) image =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) device-model-pid =3D "3255" . . . . . . . . . . . . . . . (n0,r4) serial =3D "" . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) 0 =3D "" . . . . . . . . . . . . . . . . . . . . . . . . . (n0,r4) tty =3D "/dev/pts/2" . . . . . . . . . . . . . . . . . . (n0,r4) libxl: debug: libxl_create.c:1710:do_domain_create: ao 0x803247000: = create: how=3D0x0 callback=3D0x0 poller=3D0x8032280a0 libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk = vdev=3Dxvda spec.backend=3Dunknown libxl: debug: libxl_device.c:310:disk_try_backend: Disk vdev=3Dxvda, = backend phy unsuitable due to format qcow2 libxl: debug: libxl_device.c:382:libxl__device_disk_set_backend: Disk = vdev=3Dxvda, using backend qdisk libxl: debug: libxl_create.c:970:initiate_domain_create: running = bootloader libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV = domain, skipping bootloader libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch = w=3D0x80325aab8: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline=3D"(null)", = features=3D"(null)" domainbuilder: detail: xc_dom_kernel_file: = filename=3D"/usr/local/lib/xen/boot/hvmloader" domainbuilder: detail: xc_dom_malloc_filemap : 329 kB domainbuilder: detail: xc_dom_boot_xen_init: ver 4.7, caps = xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p = hvm-3.0-x86_64=20 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...=20= domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader = ...=20 domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a = bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...=20= domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=3D0x100000 memsz=3D0x5ae04 xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15ae04 domainbuilder: detail: xc_dom_mem_init: mem 504 MB, pages 0x1f800 pages, = 4k each domainbuilder: detail: xc_dom_mem_init: 0x1f800 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: xc_dom_malloc : 1008 kB xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0000000000000200 xc: detail: 2MB PAGES: 0x00000000000000fb xc: detail: 1GB PAGES: 0x0000000000000000 domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn = 0x100+0x5b at 0x8006d0000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x100000 = -> 0x15b000 (pfn 0x100 + 0x5b pages) xc: detail: elf_load_binary: phdr 0 at 0x80072b000 -> 0x80077c3a0 domainbuilder: detail: alloc_pgtables_hvm: doing nothing domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x15b000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: = xen-3.0-x86_64 domainbuilder: detail: xc_dom_compat_check: supported guest type: = xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: = hvm-3.0-x86_32 <=3D matches domainbuilder: detail: xc_dom_compat_check: supported guest type: = hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: = hvm-3.0-x86_64 domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 1012 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 329 kB domainbuilder: detail: domU mmap : 364 kB domainbuilder: detail: vcpu_hvm: called domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff000 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk = vdev=3Dxvda spec.backend=3Dqdisk libxl: debug: libxl_device.c:1156:device_hotplug: No hotplug script to = execute libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch = w=3D0x80337a4d0: deregister unregistered libxl: debug: libxl.c:3166:libxl__device_disk_find_local_path: Directly = accessing local QDISK target /root/cirros-0.3.4-x86_64-disk.img libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: = sysconf(_SC_GETPW_R_SIZE_MAX) failed, setting the initial buffer size to = 2048 libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: = sysconf(_SC_GETPW_R_SIZE_MAX) failed, setting the initial buffer size to = 2048 libxl: debug: libxl_dm.c:1498:libxl__build_device_model_args_new: Could = not find user xen-qemuuser-shared, starting QEMU as root libxl: debug: libxl_dm.c:2092:libxl__spawn_local_dm: Spawning = device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = /usr/local/lib/xen/bin/qemu-system-i386 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -xen-domid libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 3 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = socket,id=3Dlibxl-cmd,path=3D/var/run/xen/qmp-libxl-3,server,nowait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-shutdown libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = chardev=3Dlibxl-cmd,mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = socket,id=3Dlibxenstat-cmd,path=3D/var/run/xen/qmp-libxenstat-3,server,now= ait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = chardev=3Dlibxenstat-cmd,mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -nodefaults libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-user-config libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -name libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: vm libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -vnc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 0.0.0.0:0,to=3D99 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -display libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -serial libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: pty libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -device libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = cirrus-vga,vgamem_mb=3D8 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -boot libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: order=3Dc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usb libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usbdevice libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: tablet libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -smp libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 2,maxcpus=3D2 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -net libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -machine libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: xenfv libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -m libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 504 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: = file=3D/root/cirros-0.3.4-x86_64-disk.img,if=3Dide,index=3D0,media=3Ddisk,= format=3Dqcow2,cache=3Dwriteback libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: Spawning = device-model /usr/local/lib/xen/bin/qemu-system-i386 with additional = environment: libxl: debug: libxl_dm.c:2098:libxl__spawn_local_dm: = XEN_QEMU_CONSOLE_LIMIT=3D1048576 libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch = w=3D0x80325adb0 wpath=3D/local/domain/0/device-model/3/state token=3D3/0: = register slotnum=3D3 libxl: debug: libxl_create.c:1736:do_domain_create: ao 0x803247000: = inprogress: poller=3D0x8032280a0, flags=3Di libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325adb0 = wpath=3D/local/domain/0/device-model/3/state token=3D3/0: event = epath=3D/local/domain/0/device-model/3/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 3 device model: = spawn watch p=3D(null) libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325adb0 = wpath=3D/local/domain/0/device-model/3/state token=3D3/0: event = epath=3D/local/domain/0/device-model/3/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 3 device model: = spawn watch p=3Drunning libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch = w=3D0x80325adb0 wpath=3D/local/domain/0/device-model/3/state token=3D3/0: = deregister slotnum=3D3 libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 3 = device model (dying as expected) [853] died due to fatal signal Killed libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch = w=3D0x80325adb0: deregister unregistered libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to = /var/run/xen/qmp-libxl-3 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-chardev", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-vnc", "id": 3 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_event.c:2180:libxl__ao_progress_report: ao = 0x803247000: progress report: ignored libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x803247000: = complete, rc=3D0 libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x803247000: = destroy libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to = /var/run/xen/qmp-libxl-3 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "cont", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return xencall:buffer: debug: total allocations:285 total releases:285 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:269 misses:3 toobig:13 Thanks, Alex= From owner-freebsd-xen@freebsd.org Tue Jan 24 11:45:12 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6BCBCBF8B9 for ; Tue, 24 Jan 2017 11:45:12 +0000 (UTC) (envelope-from prvs=19068b32a=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.ctxuk.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F180686B; Tue, 24 Jan 2017 11:45:08 +0000 (UTC) (envelope-from prvs=19068b32a=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.33,278,1477958400"; d="scan'208";a="39262165" Date: Tue, 24 Jan 2017 11:44:44 +0000 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Alexander Nusov CC: , Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) Message-ID: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: User-Agent: NeoMutt/20170113 (1.7.2) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2017 11:45:12 -0000 On Mon, Jan 23, 2017 at 06:25:23PM +0300, Alexander Nusov wrote: > Hello, > Sorry for cross-posting, since it's related to Xen hypervisor I'm forward= ing this message to the freebsd-xen mailing list. >=20 > I'm trying to launch a HVM DomU guest from QCOW2 image by using PV driver= on FreeBSD 11 Dom0. > The issue is that guest cannot connect to the device/vbd and requires to = wait for 4 minutes to proceed, it goes through the countdown and starts fin= e (disk, networking) >=20 > [ 6.684115] XENBUS: Waiting for devices to initialise: 25s...20s...15s= =2E..10s...5s...0s...235s...230s...225s...220s...215s...210s...205s...200s.= =2E.195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...1= 45s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...= 90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s= =2E..25s...20s...15s...10s...5s...0s... > [ 271.591403] XENBUS: Timeout connecting to device: device/vbd/51712 (lo= cal state 3, remote state 1) > [ 271.599963] XENBUS: Device with no driver: device/vkbd/0 > [ 271.604249] Magic number: 1:453:334 > ... > login:=20 >=20 >=20 > Unlike Linux It's impossible to boot FreeBSD 11 guests from QCOW2 (xenbus= b_nop_confighook_cb timeout) >=20 > Steps to reproduce: > 1. Download qcow2 cirros image (small linux)=20 > http://secure-web.cisco.com/1RtWXk7FS6UyRxe5A9ejZVfU2VK9512Yeds80mlP_0LoL= wQJLjbPBoILL2e4QyF2OaMyO8fbrQrYVmfOybSYvHgWH1sRz1gK4Zi5XDAzBDUluwhlU4NxVQ7S= 17tH6vSTrIJmBJ_NO-sdA1Ph_KfSOsNvMkw-swwuKHD9ophVFfqEe6Lnt_PIP4H-LhZfp-5_aCu= qbPYciXtMyxWbF1Od65jiFe-_LfmPRt53aCscOk7cBRI_-o603sb7htpDHH06Y8-M4oG5Lt4s1j= r1XcKTrIWhnD-Lhz55blWyc2FnCgvk/http%3A%2F%2Fdownload.cirros-cloud.net%2F0.3= =2E4%2Fcirros-0.3.4-x86_64-disk.img > > # file cirros-0.3.4-x86_64-disk.img=20 > cirros-0.3.4-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes > 2. create DomU from config bellow xl create -c config.cfg >=20 > builder =3D "hvm" > memory =3D 512 > vcpus =3D 2 > name =3D "cirros" > disk =3D [ 'file:qcow2:/root/cirros-0.3.4-x86_64-disk.img,xvda,w' ] > boot =3D "c"=20 > vnc =3D 1 > vnclisten =3D "0.0.0.0" > usbdevice =3D 'tablet' > on_poweroff =3D 'destroy' > on_reboot =3D 'restart' > on_crash =3D 'restart' > acpi =3D 1 > serial =3D 'pty' >=20 > I've also tried multiple configurations like tap:qcow2:. tap2:qcow2:, aio= :, switching from xen bus to ide. didn't work. > The only driver that had no issues was PHY but it supports only RAW image= s. >=20 > Is that a bug or I'm missing something? >=20 > tested both STABLE snapshot and 11.0-RELEASE >=20 > # uname -a > FreeBSD xen 11.0-STABLE FreeBSD 11.0-STABLE #0 r311441: Thu Jan 5 22:45:= 20 UTC 2017 >=20 > # pkg info | grep xen > xen-4.7.0_2 Xen Hypervisor meta port > xen-kernel-4.7.1_3 Hypervisor using a microkernel design > xen-tools-4.7.1_1 Xen management tool, based on LibXenlight So just that I understand this correctly, this is a FreeBSD 11.0-STABLE Dom= 0, plus the Xen packages from pkg? If that's the case, it's not going to work, FreeBSD 11.0 Dom0 doesn't yet support qcow image format for HVM/PV guests, you will have to use FreeBSD 12 (HEAD) as your Dom0, or backport r308128 into STABLE (should be self contai= ned, so I don't expect any conflicts). You will also have to apply the following patch to the xen-tools package and recompile: https://lists.freebsd.org/pipermail/freebsd-xen/2016-August/002819.html IIRC the right syntax to specify the disk device is: 'format=3Dqcow2,vdev=3Dxvda,access=3Drw,backendtype=3Dqdisk,target=3D/root/= cirros-0.3.4-x86_64-disk.img' I'm also adding Akshay to the conversation, who did the gntdev implementati= on. Roger. From owner-freebsd-xen@freebsd.org Tue Jan 24 14:45:45 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6A8ACBED61 for ; Tue, 24 Jan 2017 14:45:45 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BF059D06 for ; Tue, 24 Jan 2017 14:45:45 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1485269126; s=zoho; d=nfvexpress.com; i=alexander.nusov@nfvexpress.com; h=Date:From:To:Cc:Message-Id:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=134630; bh=H8sungo4SdpyUZ08JkA7ZYyx4ZnuY1ug/DVwV1M/zjA=; b=OCvt4RdRo2mU0G+QbUok3OXtgq5gpueGAfM6w80yYt6s+AmqjkT5ttG/Xb5fF/Qc xjWZisRfv0PBpFDsThZSLDmIA6SCJkbSbSKeDEDvF8IqlKnpL7CuOxDFfxnJlzRM58o Kf0n+GC0nJKUi3uqLPZLuBllTD8VUWU4QlEqkobM= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1485269126135750.416836660362; Tue, 24 Jan 2017 06:45:26 -0800 (PST) Date: Tue, 24 Jan 2017 17:45:25 +0300 From: Alexander Nusov To: =?UTF-8?Q?=22Roger_Pau_Monn=C3=A9=22?= Cc: , Message-Id: <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> In-Reply-To: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) MIME-Version: 1.0 X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2017 14:45:45 -0000 Yes, it was FreeBSD 11.0-STABLE Dom0 with xen-kernel/xen-tools built from t= he ports tree (head) It seems there is an issue with xen pci devices, since booting from QCOW2 i= mages actually works (even on FreeBSD 11.0-RELEASE branch) except communica= tion with /xen/vbd devices from the guest. By the way, I installed FreeBSD 12-CURRENT r311461 snapshot and applied the= patch for xen-utils and now things got worse, qemu-system-i386 process started to crash at this point: [ 1.162342] GHES: HEST is not enabled! [ 1.166829] xen-platform-pci 0000:00:02.0: PCI INT A -> GSI 24 (level= , low) -> IRQ 24 [ 1.191301] Grant table initialized [ 1.197758] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.238473] serial8250: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 1.246103] init_memory_mapping: 0000000020000000-0000000028000000 [ 1.374895] 00:0a: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 1.381290] Linux agpgart interface v0.103 [ 1.387732] brd: module loaded [ 1.392518] loop: module loaded CRASH root@current:~ # cat /var/log/xen/xl-vm.log=20 Waiting for domain vm (domid 4) to die [pid 18070] libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=3D0x803= 21c3e0 wpath=3D@releaseDomain token=3D3/0: register slotnum=3D3 libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80321c3e0 wpa= th=3D@releaseDomain token=3D3/0: event epath=3D@releaseDomain libxl: debug: libxl.c:1184:domain_death_xswatch_callback: [evg=3D0x803221aa= 0:4] nentries=3D1 rc=3D1 4..4 libxl: debug: libxl.c:1195:domain_death_xswatch_callback: [evg=3D0x803221aa= 0:4] got=3Ddomaininfos[0] got->domain=3D4 libxl: debug: libxl.c:1221:domain_death_xswatch_callback: exists shutdown_= reported=3D0 dominf.flags=3Dffff0002 libxl: debug: libxl.c:1188:domain_death_xswatch_callback: [evg=3D0] all rep= orted libxl: debug: libxl.c:1250:domain_death_xswatch_callback: domain death sear= ch done Then I did a xen-utils rollback and stuck with the same issue (Waiting for = XENBUS), no crash though. root@current:~ # cat vm.cfg=20 builder =3D "hvm"=20 memory =3D 512=20 vcpus =3D 2=20 name =3D "vm"=20 disk =3D ['format=3Dqcow2,vdev=3Dxvda,access=3Drw,backendtype=3Dqdisk,targe= t=3D/root/cirros-0.3.4-x86_64-disk.img'] boot =3D "c"=20 vnc =3D 1=20 vnclisten =3D "0.0.0.0"=20 usbdevice =3D 'tablet'=20 on_poweroff =3D 'destroy'=20 on_reboot =3D 'restart'=20 on_crash =3D 'restart'=20 acpi =3D 1=20 serial =3D 'pty' root@current:~ # cat /boot/loader.conf=20 hw.pci.mcfg=3D0 xen_kernel=3D"/boot/xen" xen_cmdline=3D"dom0_mem=3D8192M dom0_max_vcpus=3D8 dom0pvh=3D1 com1=3D11520= 0,8n1 guest_loglvl=3Dall loglvl=3Dall" root@current:~ # cat /etc/sysctl.conf=20 vm.max_wired=3D-1 root@current:~ # xl -vvv create vm.cfg Parsing config from vm.cfg libxl: debug: libxl_create.c:1710:do_domain_create: ao 0x803247000: create:= how=3D0x0 callback=3D0x0 poller=3D0x8032280a0 libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk vdev= =3Dxvda spec.backend=3Dqdisk libxl: debug: libxl_create.c:970:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV domain= , skipping bootloader libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325dab8: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline=3D"(null)", features=3D"(nu= ll)" domainbuilder: detail: xc_dom_kernel_file: filename=3D"/usr/local/lib/xen/b= oot/hvmloader" domainbuilder: detail: xc_dom_malloc_filemap : 329 kB domainbuilder: detail: xc_dom_boot_xen_init: ver 4.7, caps xen-3.0-x86_64 x= en-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64=20 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...=20 domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...= =20 domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...=20 domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=3D0x100000 memsz=3D0x5ad0c xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15ad0c domainbuilder: detail: xc_dom_mem_init: mem 504 MB, pages 0x1f800 pages, 4k= each domainbuilder: detail: xc_dom_mem_init: 0x1f800 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: xc_dom_malloc : 1008 kB xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0000000000000200 xc: detail: 2MB PAGES: 0x00000000000000fb xc: detail: 1GB PAGES: 0x0000000000000000 domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+= 0x5b at 0x8006d1000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x100000 ->= ; 0x15b000 (pfn 0x100 + 0x5b pages) xc: detail: elf_load_binary: phdr 0 at 0x80072c000 -> 0x80077d2a8 domainbuilder: detail: alloc_pgtables_hvm: doing nothing domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x15b000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x= 86_64 domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x= 86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_32 <=3D matches domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_64 domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 1012 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 329 kB domainbuilder: detail: domU mmap : 364 kB domainbuilder: detail: vcpu_hvm: called domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff000 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk vdev= =3Dxvda spec.backend=3Dqdisk libxl: debug: libxl_device.c:1156:device_hotplug: No hotplug script to exec= ute libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0337d4d0: deregister unregistered libxl: debug: libxl.c:3166:libxl__device_disk_find_local_path: Directly acc= essing local QDISK target /root/cirros-0.3.4-x86_64-disk.img libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: sysconf(_SC_GETPW_R_SI= ZE_MAX) failed, setting the initial buffer size to 2048 libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: sysconf(_SC_GETPW_R_SI= ZE_MAX) failed, setting the initial buffer size to 2048 libxl: debug: libxl_dm.c:1498:libxl__build_device_model_args_new: Could not= find user xen-qemuuser-shared, starting QEMU as root libxl: debug: libxl_dm.c:2092:libxl__spawn_local_dm: Spawning device-model = /usr/local/lib/xen/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: /usr/local/lib/xen/b= in/qemu-system-i386 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -xen-domid libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 4 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: socket,id=3Dlibxl-cm= d,path=3D/var/run/xen/qmp-libxl-4,server,nowait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-shutdown libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: chardev=3Dlibxl-cmd,= mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: socket,id=3Dlibxenst= at-cmd,path=3D/var/run/xen/qmp-libxenstat-4,server,nowait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: chardev=3Dlibxenstat= -cmd,mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -nodefaults libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-user-config libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -name libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: vm libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -vnc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 0.0.0.0:0,to=3D99 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -display libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -serial libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: pty libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -device libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: cirrus-vga,vgamem_mb= =3D8 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -boot libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: order=3Dc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usb libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usbdevice libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: tablet libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -smp libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 2,maxcpus=3D2 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -net libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -machine libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: xenfv libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -m libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 504 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: file=3D/root/cirros-= 0.3.4-x86_64-disk.img,if=3Dide,index=3D0,media=3Ddisk,format=3Dqcow2,cache= =3Dwriteback libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: Spawning device-model = /usr/local/lib/xen/bin/qemu-system-i386 with additional environment: libxl: debug: libxl_dm.c:2098:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIM= IT=3D1048576 libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=3D0x803= 25ddb0 wpath=3D/local/domain/0/device-model/4/state token=3D3/0: register s= lotnum=3D3 libxl: debug: libxl_create.c:1736:do_domain_create: ao 0x803247000: inprogr= ess: poller=3D0x8032280a0, flags=3Di libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325ddb0 wpa= th=3D/local/domain/0/device-model/4/state token=3D3/0: event epath=3D/local= /domain/0/device-model/4/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 4 device model: sp= awn watch p=3D(null) libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325ddb0 wpa= th=3D/local/domain/0/device-model/4/state token=3D3/0: event epath=3D/local= /domain/0/device-model/4/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 4 device model: sp= awn watch p=3Drunning libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325ddb0 wpath=3D/local/domain/0/device-model/4/state token=3D3/0: deregist= er slotnum=3D3 libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 4 devi= ce model (dying as expected) [18067] died due to fatal signal Killed libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325ddb0: deregister unregistered libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/= xen/qmp-libxl-4 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-chardev", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-vnc", "id": 3 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_event.c:2180:libxl__ao_progress_report: ao 0x803247000:= progress report: ignored libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x803247000: comple= te, rc=3D0 libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x803247000: destro= y libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/= xen/qmp-libxl-4 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "cont", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return xencall:buffer: debug: total allocations:289 total releases:289 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:271 misses:3 toobig:15 =E2=80=94 Thanks, Alex ---- On Tue, 24 Jan 2017 14:44:44 +0300 Roger Pau Monn=C3=A9 <roger.pau@= citrix.com> wrote ---- On Mon, Jan 23, 2017 at 06:25:23PM +0300, Alexander Nusov wrote:=20 > Hello,=20 > Sorry for cross-posting, since it's related to Xen hypervisor I'm forw= arding this message to the freebsd-xen mailing list.=20 >=20 > I'm trying to launch a HVM DomU guest from QCOW2 image by using PV dri= ver on FreeBSD 11 Dom0.=20 > The issue is that guest cannot connect to the device/vbd and requires = to wait for 4 minutes to proceed, it goes through the countdown and starts = fine (disk, networking)=20 >=20 > [ 6.684115] XENBUS: Waiting for devices to initialise: 25s...20s...15s= ...10s...5s...0s...235s...230s...225s...220s...215s...210s...205s...200s...= 195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s.= ..140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s.= ..85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...2= 5s...20s...15s...10s...5s...0s...=20 > [ 271.591403] XENBUS: Timeout connecting to device: device/vbd/51712 (= local state 3, remote state 1)=20 > [ 271.599963] XENBUS: Device with no driver: device/vkbd/0=20 > [ 271.604249] Magic number: 1:453:334=20 > ...=20 > login:=20 >=20 >=20 > Unlike Linux It's impossible to boot FreeBSD 11 guests from QCOW2 (xen= busb_nop_confighook_cb timeout)=20 >=20 > Steps to reproduce:=20 > 1. Download qcow2 cirros image (small linux)=20 > # file cirros-0.3.4-x86_64-disk.img=20 > cirros-0.3.4-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes=20 > 2. create DomU from config bellow xl create -c config.cfg=20 >=20 > builder =3D "hvm"=20 > memory =3D 512=20 > vcpus =3D 2=20 > name =3D "cirros"=20 > disk =3D [ 'file:qcow2:/root/cirros-0.3.4-x86_64-disk.img,xvda,w' ]=20 > boot =3D "c"=20 > vnc =3D 1=20 > vnclisten =3D "0.0.0.0"=20 > usbdevice =3D 'tablet'=20 > on_poweroff =3D 'destroy'=20 > on_reboot =3D 'restart'=20 > on_crash =3D 'restart'=20 > acpi =3D 1=20 > serial =3D 'pty'=20 >=20 > I've also tried multiple configurations like tap:qcow2:. tap2:qcow2:, = aio:, switching from xen bus to ide. didn't work.=20 > The only driver that had no issues was PHY but it supports only RAW im= ages.=20 >=20 > Is that a bug or I'm missing something?=20 >=20 > tested both STABLE snapshot and 11.0-RELEASE=20 >=20 > # uname -a=20 > FreeBSD xen 11.0-STABLE FreeBSD 11.0-STABLE #0 r311441: Thu Jan 5 22:4= 5:20 UTC 2017=20 >=20 > # pkg info | grep xen=20 > xen-4.7.0_2 Xen Hypervisor meta port=20 > xen-kernel-4.7.1_3 Hypervisor using a microkernel design=20 > xen-tools-4.7.1_1 Xen management tool, based on LibXenlight=20 So just that I understand this correctly, this is a FreeBSD 11.0-STABLE Dom= 0,=20 plus the Xen packages from pkg?=20 If that's the case, it's not going to work, FreeBSD 11.0 Dom0 doesn't yet= =20 support qcow image format for HVM/PV guests, you will have to use FreeBSD 1= 2=20 (HEAD) as your Dom0, or backport r308128 into STABLE (should be self contai= ned,=20 so I don't expect any conflicts). You will also have to apply the following= =20 patch to the xen-tools package and recompile:=20 https://lists.freebsd.org/pipermail/freebsd-xen/2016-August/002819.html=20 IIRC the right syntax to specify the disk device is:=20 'format=3Dqcow2,vdev=3Dxvda,access=3Drw,backendtype=3Dqdisk,target=3D/root/= cirros-0.3.4-x86_64-disk.img'=20 I'm also adding Akshay to the conversation, who did the gntdev implementati= on.=20 Roger.=20 From owner-freebsd-xen@freebsd.org Tue Jan 24 16:56:45 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82ACCBF065 for ; Tue, 24 Jan 2017 16:56:45 +0000 (UTC) (envelope-from prvs=19068b32a=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.ctxuk.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F089972; Tue, 24 Jan 2017 16:56:44 +0000 (UTC) (envelope-from prvs=19068b32a=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.33,278,1477958400"; d="scan'208";a="39293574" Date: Tue, 24 Jan 2017 16:56:21 +0000 From: =?iso-8859-1?Q?=22Roger_Pau_Monn=E9=22?= To: Alexander Nusov CC: , Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) Message-ID: <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> User-Agent: NeoMutt/20170113 (1.7.2) X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2017 16:56:46 -0000 On Tue, Jan 24, 2017 at 05:45:25PM +0300, Alexander Nusov wrote: > Yes, it was FreeBSD 11.0-STABLE Dom0 with xen-kernel/xen-tools built from the ports tree (head) > > > > It seems there is an issue with xen pci devices, since booting from QCOW2 images actually works (even on FreeBSD 11.0-RELEASE branch) except communication with /xen/vbd devices from the guest. Yes, I'm seeing exactly the same. The QEMU process is killed with a segmentation fault. Akshay, here is the full debug output: Program terminated with signal 11, Segmentation fault. [...] #0 blk_handle_requests (blkdev=0x807463c00) at hw/block/xen_disk.c:862 862 rp = blkdev->rings.common.sring->req_prod; [New Thread 8087f9000 (LWP 100947/)] [New Thread 807418800 (LWP 100945/)] [New Thread 807418300 (LWP 100944/)] [New Thread 807417e00 (LWP 100943/)] [New Thread 807417900 (LWP 100942/)] [New Thread 807417400 (LWP 100941/)] [New Thread 807416a00 (LWP 100940/)] [New Thread 807416500 (LWP 100939/)] [New Thread 807416000 (LWP 100091/)] (gdb) bt #0 blk_handle_requests (blkdev=0x807463c00) at hw/block/xen_disk.c:862 #1 0x00000000005f9dcd in blk_bh (opaque=0x807463c00) at hw/block/xen_disk.c:918 #2 0x000000000080ba69 in aio_bh_call (bh=0x80780d810) at async.c:87 #3 0x000000000080bb10 in aio_bh_poll (ctx=0x8074a0680) at async.c:115 #4 0x000000000081c099 in aio_dispatch (ctx=0x8074a0680) at aio-posix.c:303 #5 0x000000000080c2cd in aio_ctx_dispatch (source=0x8074a0680, callback=0, user_data=0x0) at async.c:254 #6 0x0000000802e3903b in g_main_context_dispatch () from /usr/local/lib/libglib-2.0.so.0 #7 0x000000000081a34c in glib_pollfds_poll () at main-loop.c:259 #8 0x0000000000819dc5 in os_host_main_loop_wait (timeout=0) at main-loop.c:306 #9 0x0000000000819c29 in main_loop_wait (nonblocking=0) at main-loop.c:556 #10 0x0000000000588ed7 in main_loop () at vl.c:1966 #11 0x0000000000583b59 in main (argc=38, argv=0x7fffffffe750, envp=0x7fffffffe888) at vl.c:4684 Current language: auto; currently minimal It seems like the device is not properly mapping the grants, and QEMU gets a SEGFAULT when trying to access the ring page. Roger. From owner-freebsd-xen@freebsd.org Wed Jan 25 10:08:33 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2F06CB59B7 for ; Wed, 25 Jan 2017 10:08:33 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-lf0-f41.google.com (mail-lf0-f41.google.com [209.85.215.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 77C55BB1 for ; Wed, 25 Jan 2017 10:08:33 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-lf0-f41.google.com with SMTP id z134so125915158lff.3 for ; Wed, 25 Jan 2017 02:08:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=T1OpzM96oI25Gg/1HDiGWSti2/IxPVrOqSIwapFRzOQ=; b=ANpIccFUAmC0o1OBAAzU+DA7rAQsVjP9/sNrlz3loeP6SiFm3lNCI4svcegbQ1fLx8 D69N1dZfBHaWNicIMnpmOMd/JTYew3EYvRQOMUGxDQj93xcxrnEi2nBS5uPf7wWICf34 seoFYtvO30ze0xeXtgTLDc0j3NCT4t4tHU4dhYjhgj5zpIo6CrWAN237hL0ZlwfSijjk ilcKOesbADddXXjxHC069nbP/e+19CspcPAtmVvldGXKYqcLHg1WlSY7X4ITG479jaEu y2tGwa6j8rKpNlepRQ0sjML2N8iC9lsK+QmvRRk2/IWA+8tvLVjSMdekDqhG8pGR0gdA Evow== X-Gm-Message-State: AIkVDXKZL0RUACcruP31VZio5XqG8wjCLA5ZIyR3JZgzZ0WK78fC660toTOLyYQO+5j5Ng== X-Received: by 10.25.20.152 with SMTP id 24mr10935224lfu.152.1485336916313; Wed, 25 Jan 2017 01:35:16 -0800 (PST) Received: from mail-lf0-f45.google.com (mail-lf0-f45.google.com. [209.85.215.45]) by smtp.gmail.com with ESMTPSA id 9sm8261967ljg.33.2017.01.25.01.35.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 01:35:15 -0800 (PST) Received: by mail-lf0-f45.google.com with SMTP id n124so125215091lfd.2 for ; Wed, 25 Jan 2017 01:35:15 -0800 (PST) X-Received: by 10.25.38.136 with SMTP id m130mr13000425lfm.90.1485336915415; Wed, 25 Jan 2017 01:35:15 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.145.68 with HTTP; Wed, 25 Jan 2017 01:35:14 -0800 (PST) Received: by 10.25.145.68 with HTTP; Wed, 25 Jan 2017 01:35:14 -0800 (PST) In-Reply-To: <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> From: Akshay Jaggi Date: Wed, 25 Jan 2017 09:35:14 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Alexander Nusov , FreeBSD-Xen Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 10:08:34 -0000 Hi Alex and Roger, Without the patch and FreeBSD 12, qcow2 doesn't actually work. We were thinking of adding code to warn against this, but then thought gntdev implementation is in process anyway. With the patch, the crash needs to be investigated (because I did run one of the gntdev based image, most probably qcow2, iirc). Sadly, I'm still searching for housing in Dublin, and hence won't be able to look at this immediately. Two weeks, to find and then move in, if I'm lucky. Meanwhile, I'll start hunting for a development machine in office. Regards, Akshay On Jan 24, 2017 16:56, Roger Pau Monn=C3=A9 wrote: > On Tue, Jan 24, 2017 at 05:45:25PM +0300, Alexander Nusov wrote: > > Yes, it was FreeBSD 11.0-STABLE Dom0 with xen-kernel/xen-tools built > from the ports tree (head) > > > > > > > > It seems there is an issue with xen pci devices, since booting from > QCOW2 images actually works (even on FreeBSD 11.0-RELEASE branch) except > communication with /xen/vbd devices from the guest. > > Yes, I'm seeing exactly the same. The QEMU process is killed with a > segmentation fault. Akshay, here is the full debug output: > > Program terminated with signal 11, Segmentation fault. > [...] > #0 blk_handle_requests (blkdev=3D0x807463c00) at hw/block/xen_disk.c:862 > 862 rp =3D blkdev->rings.common.sring->req_prod; > [New Thread 8087f9000 (LWP 100947/)] > [New Thread 807418800 (LWP 100945/)] > [New Thread 807418300 (LWP 100944/)] > [New Thread 807417e00 (LWP 100943/)] > [New Thread 807417900 (LWP 100942/)] > [New Thread 807417400 (LWP 100941/)] > [New Thread 807416a00 (LWP 100940/)] > [New Thread 807416500 (LWP 100939/)] > [New Thread 807416000 (LWP 100091/)] > (gdb) bt > #0 blk_handle_requests (blkdev=3D0x807463c00) at hw/block/xen_disk.c:862 > #1 0x00000000005f9dcd in blk_bh (opaque=3D0x807463c00) at > hw/block/xen_disk.c:918 > #2 0x000000000080ba69 in aio_bh_call (bh=3D0x80780d810) at async.c:87 > #3 0x000000000080bb10 in aio_bh_poll (ctx=3D0x8074a0680) at async.c:115 > #4 0x000000000081c099 in aio_dispatch (ctx=3D0x8074a0680) at aio-posix.c= :303 > #5 0x000000000080c2cd in aio_ctx_dispatch (source=3D0x8074a0680, > callback=3D0, user_data=3D0x0) > at async.c:254 > #6 0x0000000802e3903b in g_main_context_dispatch () from /usr/local/lib/ > libglib-2.0.so.0 > #7 0x000000000081a34c in glib_pollfds_poll () at main-loop.c:259 > #8 0x0000000000819dc5 in os_host_main_loop_wait (timeout=3D0) at > main-loop.c:306 > #9 0x0000000000819c29 in main_loop_wait (nonblocking=3D0) at main-loop.c= :556 > #10 0x0000000000588ed7 in main_loop () at vl.c:1966 > #11 0x0000000000583b59 in main (argc=3D38, argv=3D0x7fffffffe750, > envp=3D0x7fffffffe888) at vl.c:4684 > Current language: auto; currently minimal > > It seems like the device is not properly mapping the grants, and QEMU get= s > a > SEGFAULT when trying to access the ring page. > > Roger. > From owner-freebsd-xen@freebsd.org Wed Jan 25 11:21:09 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4848ECC07C3 for ; Wed, 25 Jan 2017 11:21:09 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3199718C; Wed, 25 Jan 2017 11:21:08 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1485343257; s=zoho; d=nfvexpress.com; i=alexander.nusov@nfvexpress.com; h=Date:From:To:Cc:Message-Id:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=9752; bh=s48+k4XREdQTXViyqGZv3XM6H8lNz9deQe6wONMsI5o=; b=N0qSsLIWCW17LD2gVZBF3AJp0KM47UcbDrVZ10ubmEdZcalkzo2QPVTsXRS2fkR5 jJ71ma0xjYKRhsAsusxQ4OdUcF3aN30BVZkZSabHRtdu8h0cGyrYyMMi7swhxbkVuLo HKY8xKxbCRXKw48Acrz1jf7VEUZ+NbE5Q7Dj2NfY= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1485343255500721.9455538969542; Wed, 25 Jan 2017 03:20:55 -0800 (PST) Date: Wed, 25 Jan 2017 14:20:55 +0300 From: Alexander Nusov To: "Akshay Jaggi" Cc: =?UTF-8?Q?=22Roger_Pau_Monn=C3=A9=22?= , "FreeBSD-Xen" Message-Id: <159d55b6b65.dc9b3b1720453.5847395326743424395@nfvexpress.com> In-Reply-To: References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) MIME-Version: 1.0 X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 11:21:09 -0000 Got it, thanks. I found a workaround to avoid XENBUS delay in Linux DomUs by adding xen_pla= tform_pci =3D 0 to the configuration file. So FreeBSD 11.0 Dom0 can boot Linux guests and Windows Server (FreeBSD DomU= still stuck on xenbusb_nop_confighook_cb) from QCOW2 images (also works wi= th QCOW2 overlay images) Can you tell me please what is the disadvantage of not using /dev/xen/vbd d= evices and drivers in guests? like slow I/O?=20 May it lead to data corruption? -- Alex ---- On Wed, 25 Jan 2017 12:35:14 +0300 Akshay Jaggi <jaggi@freebsd.org&= gt; wrote ---- Hi Alex and Roger, Without the patch and FreeBSD 12, qcow2 doesn't actually work. We were thin= king of adding code to warn against this, but then thought gntdev implement= ation is in process anyway. With the patch, the crash needs to be investigated (because I did run one o= f the gntdev based image, most probably qcow2, iirc). Sadly, I'm still searching for housing in Dublin, and hence won't be able t= o look at this immediately. Two weeks, to find and then move in, if I'm luc= ky. Meanwhile, I'll start hunting for a development machine in office. Regards, Akshay On Jan 24, 2017 16:56, Roger Pau Monn=C3=A9 <roger.pau@citrix.com> wr= ote: On Tue, Jan 24, 2017 at 05:45:25PM +0300, Alexander Nusov wrote: > Yes, it was FreeBSD 11.0-STABLE Dom0 with xen-kernel/xen-tools built f= rom the ports tree (head) > > > > It seems there is an issue with xen pci devices, since booting from QC= OW2 images actually works (even on FreeBSD 11.0-RELEASE branch) except comm= unication with /xen/vbd devices from the guest. Yes, I'm seeing exactly the same. The QEMU process is killed with a segmentation fault. Akshay, here is the full debug output: Program terminated with signal 11, Segmentation fault. [...] #0 blk_handle_requests (blkdev=3D0x807463c00) at hw/block/xen_disk.c:862 862 rp =3D blkdev->rings.common.sring->req_prod; [New Thread 8087f9000 (LWP 100947/<unknown>)] [New Thread 807418800 (LWP 100945/<unknown>)] [New Thread 807418300 (LWP 100944/<unknown>)] [New Thread 807417e00 (LWP 100943/<unknown>)] [New Thread 807417900 (LWP 100942/<unknown>)] [New Thread 807417400 (LWP 100941/<unknown>)] [New Thread 807416a00 (LWP 100940/<unknown>)] [New Thread 807416500 (LWP 100939/<unknown>)] [New Thread 807416000 (LWP 100091/<unknown>)] (gdb) bt #0 blk_handle_requests (blkdev=3D0x807463c00) at hw/block/xen_disk.c:862 #1 0x00000000005f9dcd in blk_bh (opaque=3D0x807463c00) at hw/block/xen_dis= k.c:918 #2 0x000000000080ba69 in aio_bh_call (bh=3D0x80780d810) at async.c:87 #3 0x000000000080bb10 in aio_bh_poll (ctx=3D0x8074a0680) at async.c:115 #4 0x000000000081c099 in aio_dispatch (ctx=3D0x8074a0680) at aio-posix.c:3= 03 #5 0x000000000080c2cd in aio_ctx_dispatch (source=3D0x8074a0680, callback= =3D0, user_data=3D0x0) at async.c:254 #6 0x0000000802e3903b in g_main_context_dispatch () from /usr/local/lib/li= bglib-2.0.so.0 #7 0x000000000081a34c in glib_pollfds_poll () at main-loop.c:259 #8 0x0000000000819dc5 in os_host_main_loop_wait (timeout=3D0) at main-loop= .c:306 #9 0x0000000000819c29 in main_loop_wait (nonblocking=3D0) at main-loop.c:5= 56 #10 0x0000000000588ed7 in main_loop () at vl.c:1966 #11 0x0000000000583b59 in main (argc=3D38, argv=3D0x7fffffffe750, envp=3D0x= 7fffffffe888) at vl.c:4684 Current language: auto; currently minimal It seems like the device is not properly mapping the grants, and QEMU gets = a SEGFAULT when trying to access the ring page. Roger. From owner-freebsd-xen@freebsd.org Wed Jan 25 11:51:05 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12944CBF277 for ; Wed, 25 Jan 2017 11:51:05 +0000 (UTC) (envelope-from prvs=19161f0f8=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.ctxuk.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 63F741DC; Wed, 25 Jan 2017 11:51:03 +0000 (UTC) (envelope-from prvs=19161f0f8=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.33,283,1477958400"; d="scan'208";a="39347490" Date: Wed, 25 Jan 2017 11:50:51 +0000 From: =?iso-8859-1?Q?=22Roger_Pau_Monn=E9=22?= To: Alexander Nusov CC: Akshay Jaggi , FreeBSD-Xen Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) Message-ID: <20170125115051.sasgqxwgt47p7pwi@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> <159d55b6b65.dc9b3b1720453.5847395326743424395@nfvexpress.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <159d55b6b65.dc9b3b1720453.5847395326743424395@nfvexpress.com> User-Agent: NeoMutt/20170113 (1.7.2) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 11:51:05 -0000 On Wed, Jan 25, 2017 at 02:20:55PM +0300, Alexander Nusov wrote: > Got it, thanks. > > > > I found a workaround to avoid XENBUS delay in Linux DomUs by adding xen_platform_pci = 0 to the configuration file. > > So FreeBSD 11.0 Dom0 can boot Linux guests and Windows Server (FreeBSD DomU still stuck on xenbusb_nop_confighook_cb) from QCOW2 images (also works with QCOW2 overlay images) > On FreeBSD you can boot with QCOW2 by not setting "xen_platform_pci = 0" on the guest config file and adding "hw.xen.disable_pv_disks=1" to the /boot/loader.conf file of the guest. This will prevent FreeBSD from using the PV disk, but you will still get the PV network interfaces. > > Can you tell me please what is the disadvantage of not using /dev/xen/vbd devices and drivers in guests? like slow I/O? Slow IO only. > May it lead to data corruption? No. From owner-freebsd-xen@freebsd.org Wed Jan 25 11:58:28 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B75BECBF4E3 for ; Wed, 25 Jan 2017 11:58:28 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A036C87E; Wed, 25 Jan 2017 11:58:28 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1485345501; s=zoho; d=nfvexpress.com; i=alexander.nusov@nfvexpress.com; h=Date:From:To:Cc:Message-Id:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=3517; bh=wqnR98HUL6+LWjuzrBc6jyUN1yDZV+19ylX7gfHL2+o=; b=YjWqoZhbhieCF3pwZU6P1Mc2bRGY3HlFCmqn/b4hRB/SZvgRFP1AGodDZ8UMBTnG iHY2lXEDpwTAXQMG3gw6N0rjY4GTMMFbriS2yuWcia6pdjplqP1hPzGXKSAIGSnKOm6 4GVxTPDQG0SWxu7aG9wFvc4TX5JWwYNcsuIuvbd4= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1485345499407371.2166860166391; Wed, 25 Jan 2017 03:58:19 -0800 (PST) Date: Wed, 25 Jan 2017 14:58:19 +0300 From: Alexander Nusov To: =?UTF-8?Q?=22Roger_Pau_Monn=C3=A9=22?= Cc: "Akshay Jaggi" , "FreeBSD-Xen" Message-Id: <159d57da8bb.e9fea5dc25161.6113276104371162720@nfvexpress.com> In-Reply-To: <20170125115051.sasgqxwgt47p7pwi@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> <20170124165621.iidjypfoyp4ccysi@dhcp-3-221.uk.xensource.com> <159d55b6b65.dc9b3b1720453.5847395326743424395@nfvexpress.com> <20170125115051.sasgqxwgt47p7pwi@dhcp-3-221.uk.xensource.com> Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) MIME-Version: 1.0 X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 11:58:28 -0000 Cool, many thanks! -- Alex ---- On Wed, 25 Jan 2017 14:50:51 +0300 Roger Pau Monn=C3=A9 <roger.pau@= citrix.com> wrote ---- On Wed, Jan 25, 2017 at 02:20:55PM +0300, Alexander Nusov wrote:=20 > Got it, thanks.=20 >=20 >=20 >=20 > I found a workaround to avoid XENBUS delay in Linux DomUs by adding xe= n_platform_pci =3D 0 to the configuration file.=20 >=20 > So FreeBSD 11.0 Dom0 can boot Linux guests and Windows Server (FreeBSD= DomU still stuck on xenbusb_nop_confighook_cb) from QCOW2 images (also wor= ks with QCOW2 overlay images)=20 >=20 =20 On FreeBSD you can boot with QCOW2 by not setting "xen_platform_pci =3D 0" = on the=20 guest config file and adding "hw.xen.disable_pv_disks=3D1" to the=20 /boot/loader.conf file of the guest. This will prevent FreeBSD from using t= he=20 PV disk, but you will still get the PV network interfaces.=20 =20 >=20 > Can you tell me please what is the disadvantage of not using /dev/xen/= vbd devices and drivers in guests? like slow I/O?=20 =20 Slow IO only.=20 =20 > May it lead to data corruption?=20 =20 No.=20