From owner-freebsd-xen@freebsd.org Tue Jan 24 14:45:45 2017 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6A8ACBED61 for ; Tue, 24 Jan 2017 14:45:45 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BF059D06 for ; Tue, 24 Jan 2017 14:45:45 +0000 (UTC) (envelope-from alexander.nusov@nfvexpress.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1485269126; s=zoho; d=nfvexpress.com; i=alexander.nusov@nfvexpress.com; h=Date:From:To:Cc:Message-Id:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=134630; bh=H8sungo4SdpyUZ08JkA7ZYyx4ZnuY1ug/DVwV1M/zjA=; b=OCvt4RdRo2mU0G+QbUok3OXtgq5gpueGAfM6w80yYt6s+AmqjkT5ttG/Xb5fF/Qc xjWZisRfv0PBpFDsThZSLDmIA6SCJkbSbSKeDEDvF8IqlKnpL7CuOxDFfxnJlzRM58o Kf0n+GC0nJKUi3uqLPZLuBllTD8VUWU4QlEqkobM= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1485269126135750.416836660362; Tue, 24 Jan 2017 06:45:26 -0800 (PST) Date: Tue, 24 Jan 2017 17:45:25 +0300 From: Alexander Nusov To: =?UTF-8?Q?=22Roger_Pau_Monn=C3=A9=22?= Cc: , Message-Id: <159d0f04b55.10bbf935114648.7927688075504705395@nfvexpress.com> In-Reply-To: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> References: <20170124114444.xdl3qj35lwebkso7@dhcp-3-221.uk.xensource.com> Subject: Re: Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb) MIME-Version: 1.0 X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2017 14:45:45 -0000 Yes, it was FreeBSD 11.0-STABLE Dom0 with xen-kernel/xen-tools built from t= he ports tree (head) It seems there is an issue with xen pci devices, since booting from QCOW2 i= mages actually works (even on FreeBSD 11.0-RELEASE branch) except communica= tion with /xen/vbd devices from the guest. By the way, I installed FreeBSD 12-CURRENT r311461 snapshot and applied the= patch for xen-utils and now things got worse, qemu-system-i386 process started to crash at this point: [ 1.162342] GHES: HEST is not enabled! [ 1.166829] xen-platform-pci 0000:00:02.0: PCI INT A -> GSI 24 (level= , low) -> IRQ 24 [ 1.191301] Grant table initialized [ 1.197758] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.238473] serial8250: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 1.246103] init_memory_mapping: 0000000020000000-0000000028000000 [ 1.374895] 00:0a: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 1.381290] Linux agpgart interface v0.103 [ 1.387732] brd: module loaded [ 1.392518] loop: module loaded CRASH root@current:~ # cat /var/log/xen/xl-vm.log=20 Waiting for domain vm (domid 4) to die [pid 18070] libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=3D0x803= 21c3e0 wpath=3D@releaseDomain token=3D3/0: register slotnum=3D3 libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80321c3e0 wpa= th=3D@releaseDomain token=3D3/0: event epath=3D@releaseDomain libxl: debug: libxl.c:1184:domain_death_xswatch_callback: [evg=3D0x803221aa= 0:4] nentries=3D1 rc=3D1 4..4 libxl: debug: libxl.c:1195:domain_death_xswatch_callback: [evg=3D0x803221aa= 0:4] got=3Ddomaininfos[0] got->domain=3D4 libxl: debug: libxl.c:1221:domain_death_xswatch_callback: exists shutdown_= reported=3D0 dominf.flags=3Dffff0002 libxl: debug: libxl.c:1188:domain_death_xswatch_callback: [evg=3D0] all rep= orted libxl: debug: libxl.c:1250:domain_death_xswatch_callback: domain death sear= ch done Then I did a xen-utils rollback and stuck with the same issue (Waiting for = XENBUS), no crash though. root@current:~ # cat vm.cfg=20 builder =3D "hvm"=20 memory =3D 512=20 vcpus =3D 2=20 name =3D "vm"=20 disk =3D ['format=3Dqcow2,vdev=3Dxvda,access=3Drw,backendtype=3Dqdisk,targe= t=3D/root/cirros-0.3.4-x86_64-disk.img'] boot =3D "c"=20 vnc =3D 1=20 vnclisten =3D "0.0.0.0"=20 usbdevice =3D 'tablet'=20 on_poweroff =3D 'destroy'=20 on_reboot =3D 'restart'=20 on_crash =3D 'restart'=20 acpi =3D 1=20 serial =3D 'pty' root@current:~ # cat /boot/loader.conf=20 hw.pci.mcfg=3D0 xen_kernel=3D"/boot/xen" xen_cmdline=3D"dom0_mem=3D8192M dom0_max_vcpus=3D8 dom0pvh=3D1 com1=3D11520= 0,8n1 guest_loglvl=3Dall loglvl=3Dall" root@current:~ # cat /etc/sysctl.conf=20 vm.max_wired=3D-1 root@current:~ # xl -vvv create vm.cfg Parsing config from vm.cfg libxl: debug: libxl_create.c:1710:do_domain_create: ao 0x803247000: create:= how=3D0x0 callback=3D0x0 poller=3D0x8032280a0 libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk vdev= =3Dxvda spec.backend=3Dqdisk libxl: debug: libxl_create.c:970:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV domain= , skipping bootloader libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325dab8: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline=3D"(null)", features=3D"(nu= ll)" domainbuilder: detail: xc_dom_kernel_file: filename=3D"/usr/local/lib/xen/b= oot/hvmloader" domainbuilder: detail: xc_dom_malloc_filemap : 329 kB domainbuilder: detail: xc_dom_boot_xen_init: ver 4.7, caps xen-3.0-x86_64 x= en-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64=20 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...=20 domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...= =20 domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...=20 domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=3D0x100000 memsz=3D0x5ad0c xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15ad0c domainbuilder: detail: xc_dom_mem_init: mem 504 MB, pages 0x1f800 pages, 4k= each domainbuilder: detail: xc_dom_mem_init: 0x1f800 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: xc_dom_malloc : 1008 kB xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0000000000000200 xc: detail: 2MB PAGES: 0x00000000000000fb xc: detail: 1GB PAGES: 0x0000000000000000 domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+= 0x5b at 0x8006d1000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x100000 ->= ; 0x15b000 (pfn 0x100 + 0x5b pages) xc: detail: elf_load_binary: phdr 0 at 0x80072c000 -> 0x80077d2a8 domainbuilder: detail: alloc_pgtables_hvm: doing nothing domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x15b000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x= 86_64 domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x= 86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_32 <=3D matches domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x= 86_64 domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 1012 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 329 kB domainbuilder: detail: domU mmap : 364 kB domainbuilder: detail: vcpu_hvm: called domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff000 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=3D0xff001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:347:libxl__device_disk_set_backend: Disk vdev= =3Dxvda spec.backend=3Dqdisk libxl: debug: libxl_device.c:1156:device_hotplug: No hotplug script to exec= ute libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0337d4d0: deregister unregistered libxl: debug: libxl.c:3166:libxl__device_disk_find_local_path: Directly acc= essing local QDISK target /root/cirros-0.3.4-x86_64-disk.img libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: sysconf(_SC_GETPW_R_SI= ZE_MAX) failed, setting the initial buffer size to 2048 libxl: debug: libxl_dm.c:755:libxl__dm_runas_helper: sysconf(_SC_GETPW_R_SI= ZE_MAX) failed, setting the initial buffer size to 2048 libxl: debug: libxl_dm.c:1498:libxl__build_device_model_args_new: Could not= find user xen-qemuuser-shared, starting QEMU as root libxl: debug: libxl_dm.c:2092:libxl__spawn_local_dm: Spawning device-model = /usr/local/lib/xen/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: /usr/local/lib/xen/b= in/qemu-system-i386 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -xen-domid libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 4 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: socket,id=3Dlibxl-cm= d,path=3D/var/run/xen/qmp-libxl-4,server,nowait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-shutdown libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: chardev=3Dlibxl-cmd,= mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: socket,id=3Dlibxenst= at-cmd,path=3D/var/run/xen/qmp-libxenstat-4,server,nowait libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: chardev=3Dlibxenstat= -cmd,mode=3Dcontrol libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -nodefaults libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -no-user-config libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -name libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: vm libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -vnc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 0.0.0.0:0,to=3D99 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -display libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -serial libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: pty libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -device libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: cirrus-vga,vgamem_mb= =3D8 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -boot libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: order=3Dc libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usb libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -usbdevice libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: tablet libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -smp libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 2,maxcpus=3D2 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -net libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -machine libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: xenfv libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -m libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: 504 libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: file=3D/root/cirros-= 0.3.4-x86_64-disk.img,if=3Dide,index=3D0,media=3Ddisk,format=3Dqcow2,cache= =3Dwriteback libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: Spawning device-model = /usr/local/lib/xen/bin/qemu-system-i386 with additional environment: libxl: debug: libxl_dm.c:2098:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIM= IT=3D1048576 libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=3D0x803= 25ddb0 wpath=3D/local/domain/0/device-model/4/state token=3D3/0: register s= lotnum=3D3 libxl: debug: libxl_create.c:1736:do_domain_create: ao 0x803247000: inprogr= ess: poller=3D0x8032280a0, flags=3Di libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325ddb0 wpa= th=3D/local/domain/0/device-model/4/state token=3D3/0: event epath=3D/local= /domain/0/device-model/4/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 4 device model: sp= awn watch p=3D(null) libxl: debug: libxl_event.c:573:watchfd_callback: watch w=3D0x80325ddb0 wpa= th=3D/local/domain/0/device-model/4/state token=3D3/0: event epath=3D/local= /domain/0/device-model/4/state libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 4 device model: sp= awn watch p=3Drunning libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325ddb0 wpath=3D/local/domain/0/device-model/4/state token=3D3/0: deregist= er slotnum=3D3 libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 4 devi= ce model (dying as expected) [18067] died due to fatal signal Killed libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=3D0x8= 0325ddb0: deregister unregistered libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/= xen/qmp-libxl-4 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-chardev", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "query-vnc", "id": 3 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_event.c:2180:libxl__ao_progress_report: ao 0x803247000:= progress report: ignored libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x803247000: comple= te, rc=3D0 libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x803247000: destro= y libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/= xen/qmp-libxl-4 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{ "execute": "cont", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return xencall:buffer: debug: total allocations:289 total releases:289 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:271 misses:3 toobig:15 =E2=80=94 Thanks, Alex ---- On Tue, 24 Jan 2017 14:44:44 +0300 Roger Pau Monn=C3=A9 <roger.pau@= citrix.com> wrote ---- On Mon, Jan 23, 2017 at 06:25:23PM +0300, Alexander Nusov wrote:=20 > Hello,=20 > Sorry for cross-posting, since it's related to Xen hypervisor I'm forw= arding this message to the freebsd-xen mailing list.=20 >=20 > I'm trying to launch a HVM DomU guest from QCOW2 image by using PV dri= ver on FreeBSD 11 Dom0.=20 > The issue is that guest cannot connect to the device/vbd and requires = to wait for 4 minutes to proceed, it goes through the countdown and starts = fine (disk, networking)=20 >=20 > [ 6.684115] XENBUS: Waiting for devices to initialise: 25s...20s...15s= ...10s...5s...0s...235s...230s...225s...220s...215s...210s...205s...200s...= 195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s.= ..140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s.= ..85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...2= 5s...20s...15s...10s...5s...0s...=20 > [ 271.591403] XENBUS: Timeout connecting to device: device/vbd/51712 (= local state 3, remote state 1)=20 > [ 271.599963] XENBUS: Device with no driver: device/vkbd/0=20 > [ 271.604249] Magic number: 1:453:334=20 > ...=20 > login:=20 >=20 >=20 > Unlike Linux It's impossible to boot FreeBSD 11 guests from QCOW2 (xen= busb_nop_confighook_cb timeout)=20 >=20 > Steps to reproduce:=20 > 1. Download qcow2 cirros image (small linux)=20 > # file cirros-0.3.4-x86_64-disk.img=20 > cirros-0.3.4-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes=20 > 2. create DomU from config bellow xl create -c config.cfg=20 >=20 > builder =3D "hvm"=20 > memory =3D 512=20 > vcpus =3D 2=20 > name =3D "cirros"=20 > disk =3D [ 'file:qcow2:/root/cirros-0.3.4-x86_64-disk.img,xvda,w' ]=20 > boot =3D "c"=20 > vnc =3D 1=20 > vnclisten =3D "0.0.0.0"=20 > usbdevice =3D 'tablet'=20 > on_poweroff =3D 'destroy'=20 > on_reboot =3D 'restart'=20 > on_crash =3D 'restart'=20 > acpi =3D 1=20 > serial =3D 'pty'=20 >=20 > I've also tried multiple configurations like tap:qcow2:. tap2:qcow2:, = aio:, switching from xen bus to ide. didn't work.=20 > The only driver that had no issues was PHY but it supports only RAW im= ages.=20 >=20 > Is that a bug or I'm missing something?=20 >=20 > tested both STABLE snapshot and 11.0-RELEASE=20 >=20 > # uname -a=20 > FreeBSD xen 11.0-STABLE FreeBSD 11.0-STABLE #0 r311441: Thu Jan 5 22:4= 5:20 UTC 2017=20 >=20 > # pkg info | grep xen=20 > xen-4.7.0_2 Xen Hypervisor meta port=20 > xen-kernel-4.7.1_3 Hypervisor using a microkernel design=20 > xen-tools-4.7.1_1 Xen management tool, based on LibXenlight=20 So just that I understand this correctly, this is a FreeBSD 11.0-STABLE Dom= 0,=20 plus the Xen packages from pkg?=20 If that's the case, it's not going to work, FreeBSD 11.0 Dom0 doesn't yet= =20 support qcow image format for HVM/PV guests, you will have to use FreeBSD 1= 2=20 (HEAD) as your Dom0, or backport r308128 into STABLE (should be self contai= ned,=20 so I don't expect any conflicts). You will also have to apply the following= =20 patch to the xen-tools package and recompile:=20 https://lists.freebsd.org/pipermail/freebsd-xen/2016-August/002819.html=20 IIRC the right syntax to specify the disk device is:=20 'format=3Dqcow2,vdev=3Dxvda,access=3Drw,backendtype=3Dqdisk,target=3D/root/= cirros-0.3.4-x86_64-disk.img'=20 I'm also adding Akshay to the conversation, who did the gntdev implementati= on.=20 Roger.=20