Date: Thu, 24 Aug 2017 19:51:58 -0700 From: Kirk McKusick <mckusick@mckusick.com> To: Aijaz Baig <aijazbaig1@gmail.com> Cc: jhb@freebsd.org, Julian Elischer <julian@freebsd.org>, freebsd-fs@freebsd.org Subject: Re: Tips on remote debugging for filesystem code Message-ID: <201708250251.v7P2pwcl029687@chez.mckusick.com> In-Reply-To: <CAHB2L%2BcbBTfT%2Bpvg=c_39EKBSoLAo-_2KdmCcYgURQK5Gio4Xg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> From: Aijaz Baig <aijazbaig1@gmail.com> > Date: Fri, 25 Aug 2017 06:33:25 +0530 > Subject: Re: Tips on remote debugging for filesystem code > To: Kirk McKusick <mckusick@mckusick.com> > Cc: Julian Elischer <julian@freebsd.org>, freebsd-fs@freebsd.org > X-ASK-Info: Message Queued (2017/08/24 18:09:30) > X-ASK-Info: Confirmed by User (2017/08/24 18:40:26) > = > On Fri, Aug 25, 2017 at 12:00 AM, Kirk McKusick <mckusick@mckusick.com> = wrote: > = >>> To: Aijaz Baig <aijazbaig1@gmail.com>, freebsd-fs@freebsd.org >>> Subject: Re: Tips on remote debugging for filesystem code >>> From: Julian Elischer <julian@freebsd.org> >>> Date: Thu, 24 Aug 2017 16:24:50 +0800 >>> >>> On 24/8/17 1:40 pm, Aijaz Baig wrote: >>> >>>> ... >>>> How do you guys normally do it? Keen to hear tips and best practices >>> >>> I have had more success recently using BHype to make a Virtual FreeBSD >>> machine and connecting to it using the built-in gdb interface. >>> >>> Firstly it is easier than a serial interface and secondly you don't >>> need two machines. >> >> Is there documentation that describes how to do this? I have been using >> a patched up set of scripts provided by John Baldwin a couple of years >> ago to use kgdb with a bhyve VM. But if there is an existing way to do >> it now, I would rather switch to it. >> >> Kirk McKusick > = > Yes please I would love to hear about this as well. Nonetheless if > any one of you Kirk or Julian could help me get started a bit with > what how you guys use it (and those scripts that Kirk talks about), > it would be really nice. Below I enclose an email from John Baldwin that got me going on debugging under bhyve. I am copying John as he likely has progressed considerably in the last two years, so can likely provide some more up to date tips. Though as far as I can see, he never added his updates to the bhyve/vmrun.sh script. So I am not sure how Julian has set up his bhyve debugging which is what lead to my initial question. > By the way is this bhyve the only way to go for fs code debugging? > = > Best Regards, > Aijaz Baig You could use multiple machines or run under other hypervisors, but bhyve lets you do everything on one machine and has the benefit of easily integrating with kgdb. Another option is to read the several chapters in my book that describes how the VFS layer works ;-) Kirk McKusick =3D-=3D-=3D From: John Baldwin <jhb@freebsd.org> Date: Tue, 08 Sep 2015 16:53:26 -0700 To: Kirk McKusick <mckusick@mckusick.com> Subject: Re: kgdb ported to devel/gdb On Tuesday, September 08, 2015 04:05:06 PM Kirk McKusick wrote: >> From: John Baldwin <jhb@freebsd.org> >> To: freebsd-current@freebsd.org >> Subject: kgdb ported to devel/gdb >> Date: Mon, 31 Aug 2015 14:32:04 -0700 >> = >> Over the past several months I have ported kgdb to the version of >> gdb in ports. I have a pending patch to the gdb port to add fork >> following, but once that is done (and possibly after updating to >> 7.10) I will try to add my existing work as a KGDB option on the >> port. Until such time, you can try the newer kgdb by checking out >> my branch from git. >> = >> Here's my cheat sheet on how to build the newer kgdb. Note that >> if you build a world with my cross-libkvm patches you should get a >> kgdb that can debug i386 cores on amd64 and vice versa. >> = >> All of the targets that the native devel/gdb support have their >> backends ported (so x86, sparc64, powerpc and powerpc64). I have >> not yet ported arm or mips since those don't work for userland yet >> in upstream gdb. I have only compiled non-x86 backends. Testing >> of the new kgdb on sparc64 and powerpc would be appreciated. >> = >> Steps: >> = >> % git clone https://github.com/bsdjhb/gdb.git >> % git checkout freebsd-7.9.1-kgdb >> % fetch http://www.freebsd.org/~jhb/gdb/build >> % pkg install devel/gdb >> = >> # Having gdb installed will mean you get the python bindings in the rig= ht >> # place. >> = >> % pkg install gmake >> = >> # I think this is the only build tool you need? >> = >> % ./build >> % cd obj >> = >> # Replace 'obj' with 'obj.<arch>' for all but amd64 >> = >> % gmake >> = >> # ... wait >> = >> You will now have a binary at 'obj/gdb/kgdb'. I just run it from my ob= j >> tree currently when testing. Once it becomes part of the port it will = get >> installed as /usr/local/bin/kgdb791 or some such. > = > This is very helpful, thanks for doing it. > = > I recently dumped my (very ancient) i386 debugging machine and started > debugging in a bhyve virtual machine. Much quicker to boot and test. > But, I lost the ability to remote debug over a serial line. Is there > some way to remote attach to my virtual machine so I can once again > set breakpoints, single step it, etc? Yes, I do this with my VMs (and this is how I tested the i386 <--> amd64 cross debugging). The way I do this is to attach a nmdm device to COM2 (uart1) in my VMs and then run gdb over that. First, I use a patched vmru= n.sh to always create a /dev/nmdm<vmname>2{A,B} pair of devices attached to COM= 2 on each VM. The 'B' device is used by bhyve, and I use the 'A' device on = the host with 'target remote' in kgdb. My change to vmrun.sh is this: --- /usr/share/examples/bhyve/vmrun.sh 2015-08-15 16:31:57.891092000 -070= 0 +++ bhyve/vmrun.sh 2015-09-08 16:46:05.000000000 -0700 @@ -91,9 +91,13 @@ loader_opt=3D"" bhyverun_opt=3D"-H -A -P" pass_total=3D0 +com2=3D"" -while getopts ac:C:d:e:g:hH:iI:m:p:t: c ; do +while getopts 2:ac:C:d:e:g:hH:iI:m:p:t: c ; do case $c in + 2) + com2=3D${OPTARG} + ;; a) bhyverun_opt=3D"${bhyverun_opt} -a" ;; @@ -269,7 +273,13 @@ i=3D$(($i + 1)) done + if [ -n "${com2}" ]; then + devargs=3D"$devargs -l com2,${com2}" + elif kldstat -qm nmdm; then + devargs=3D"$devargs -l com2,/dev/nmdm${vmname}2B" + fi + ${FBSDRUN} -c ${cpus} -m ${memsize} ${bhyverun_opt} \ -g ${gdbport} \ -s 0:0,hostbridge \ -s 1:0,lpc \ Note that if you have nmdm kldloaded, this version will place a nmdm device on COM2 automatically. Next, I enable gdb on COM2 by adding 'hint.uart.1.flags=3D"0x80"' to /boot/device.hints inside the VM. To test it, I boot the VM and then run 'sysctl debug.kdb.enter=3D1' on the console (or over an ssh session). On the host I run my new kgdb (since it can do i386 cross debugging via remote out of the box). Note that you will need the kernel.debug on the host. I do this by keeping my work tree on my host machine and exporting it via NFS into the guest. I then build the kernel the "old" way using config and make directly so that the kernel.debug is at 'sys/i386/compile/<FOO>/kernel.debug'. You can then run the new kgdb as 'kgdb /path/to/kernel.debug'. In the VM console, run 'gdb' at the db> prompt to switch over to GDB. You can then use 'target remote /dev/nmdm<vmname>2A' to connect to it. For example, if your VM is named vm0, you would use /dev/nmdmvm02A. -- = John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201708250251.v7P2pwcl029687>