From owner-freebsd-virtualization@FreeBSD.ORG Tue May 6 04:45:42 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADC698C4; Tue, 6 May 2014 04:45:42 +0000 (UTC) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 67011F3B; Tue, 6 May 2014 04:45:42 +0000 (UTC) Received: from torb.pix.net (torb.pix.net [IPv6:2001:470:e254:10:12dd:b1ff:febf:eca9]) (authenticated bits=0) by hydra.pix.net (8.14.5/8.14.5) with ESMTP id s464jeqw061729; Tue, 6 May 2014 00:45:40 -0400 (EDT) (envelope-from lidl@pix.net) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.98 at mail.pix.net Message-ID: <536868F4.7010006@pix.net> Date: Tue, 06 May 2014 00:45:40 -0400 From: Kurt Lidl User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Peter Grehan Subject: Re: Problems with bhyve's kgdb support and loadable modules References: <5363F399.40709@pix.net> <5365AD6C.5060107@freebsd.org> In-Reply-To: <5365AD6C.5060107@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2014 04:45:42 -0000 On 5/3/14 11:01 PM, Peter Grehan wrote: > Hi Kurt, > >> Is there any reasonable tutorial for using kgdb with the >> bvmdebug kernel option? > > Not really (https://wiki.freebsd.org/BHyVe/gdb) > > In any event, 9.2 doesn't have bvmdebug, though it would be a simple > backport. It's also not strictly required - you can use the serial port > same as on h/w; see below. > >> A couple of folks I know have run into issues trying to >> debug a FreeBSD stable/9 kernel from their bhyve >> hosting machine (running stable/10). >> >> In particular, the loadable modules that are in use in >> the stable/9 kernel are being "troublesome" to get to >> the point where source-level debugging actually works. >> >> Even a pointer to a couple of "worked" examples might be >> useful. >> >> I've read this: >> http://people.freebsd.org/~jhb/papers/bsdcan/2008/article/node4.html >> but not all the techniques in there appear to work properly. > > I tried to repro this with some success from a host running CURRENT. > > Firstly, I installed a 9.2 VM, with source. I edited GENERIC and added > options DDB and GDB, and reinstalled the kernel. > > The disk was then copied, and mdconfig'd/mounted on the host to > provide access to the just-buit 9.2 kernel syms and sources. > > com2 was set up as a debug port by dropping to the bhyveload prompt and > > hint.uart.1.flags="0x80" > > (this could also have been done in the guest's /boot/loader.conf) > > com2 was then set up in the bhyve command line to point to an nmdm device > > -l com2,/dev/nmdm0A > > The guest probed uart1 as a debug port: > > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x80 on acpi0 > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 64 > uart1: fast interrupt > uart1: debug port (9600,n,8,1) > > After booting, I loaded the the tap device in the guest to provide a > kld for kgdb to examine: > > root@fbsd9-2:~ # kldload if_tap > root@fbsd9-2:~ # kldstat > Id Refs Address Size Name > 1 3 0xffffffff80200000 15f92d8 kernel > 2 1 0xffffffff81a12000 59e9 if_tap.ko > > Now time to try kgdb: > > root@fbsd9-2:~ # sysctl debug.kdb.enter=1 > debug.kdb.enter: 0KDB: enter: sysctl debug.kdb.enter > [ thread pid 577 tid 100054 ] > Stopped at kdb_enter+0x3b: movq $0,0xaf0362(%rip) > db> gdb > (ctrl-c will return control to ddb) > Switching to gdb back-end > Switching to gdb back-end > > In another window, cd'd to the mounted copy of the 9.2 disk: > > kgdb -r /dev/nmdm0B kernel.debug > ... > This GDB was configured as "amd64-marcel-freebsd"...Switching to remote > protocol > kdb_enter (why=0xffffffff80fbf671 "sysctl", msg=0x80
of bounds>) at ../../../kern/subr_kdb.c:441 > 441 kdb_why = KDB_WHY_UNSET; > > (kgdb) > > After some experimentation, I found the way to get the correct symbols > for the kld was to manually specify it: > > (kgdb) add-kld /mnt/boot/kernel/if_tap.ko > add symbol table from file "/mnt/boot/kernel/if_tap.ko.symbols" at > .text_addr = 0xffffffff81a12000 > .rodata.str1.8_addr = 0xffffffff81a13b50 > .rodata.str1.1_addr = 0xffffffff81a13ddb > set_sysinit_set_addr = 0xffffffff81a13f68 > set_modmetadata_set_addr = 0xffffffff81a13f80 > set_sysctl_set_addr = 0xffffffff81a13f90 > set_sysuninit_set_addr = 0xffffffff81a13fc0 > .data_addr = 0xffffffff81a13fe0 > .bss_addr = 0xffffffff81a14620 > (y or n) y > > However, 'info sharedlibrary' didn't seem to reflect this: > > (kgdb) info sharedlibrary > From To Syms Read Shared Object Library > 0xffffffff81a12000 0xffffffff81a13c04 No /boot/kernel/if_tap.ko.symbols > > Might have been a bug there, since I was able to successfully set > breakpoints in if_tap routines and have them trigger. > > I did have some trouble getting the source path set up correctly, but > never fully investigated that: seems like gdb is quite rich in that area > and it should be possible to get sorted. > > later, > > Peter. > > > Thanks for your help. It showed me one or two things that I didn't know before, and confirms some of the shortcomings of the current kgdb that others have noticed in trying to get this work. Namely, that the 'info sharedlibrary' command doesn't seem to work properly. -Kurt