Date: Sun, 27 Aug 2017 22:04:41 +0530 From: Aijaz Baig <aijazbaig1@gmail.com> To: Kirk McKusick <mckusick@mckusick.com> Cc: jhb@freebsd.org, Julian Elischer <julian@freebsd.org>, freebsd-fs@freebsd.org Subject: Re: Tips on remote debugging for filesystem code Message-ID: <CAHB2L%2BcrbMGvwAB7zx%2BnaLC5DTbOy7RWbxyFyPCGAZmaLoQ5fg@mail.gmail.com> In-Reply-To: <201708250251.v7P2pwcl029687@chez.mckusick.com> References: <CAHB2L%2BcbBTfT%2Bpvg=c_39EKBSoLAo-_2KdmCcYgURQK5Gio4Xg@mail.gmail.com> <201708250251.v7P2pwcl029687@chez.mckusick.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks for the heads up Kirk. Unfortunately I just ran into a roadblock with Bhyve since I am running FreeBSD on core2duo which "officially" doesn't support Bhyve yet. Is there some other way I can get this to work? I have installed FreeBSD bare metal now if it helps. On Fri, Aug 25, 2017 at 8:21 AM, Kirk McKusick <mckusick@mckusick.com> wrote: > > From: Aijaz Baig <aijazbaig1@gmail.com> > > Date: Fri, 25 Aug 2017 06:33:25 +0530 > > Subject: Re: Tips on remote debugging for filesystem code > > To: Kirk McKusick <mckusick@mckusick.com> > > Cc: Julian Elischer <julian@freebsd.org>, freebsd-fs@freebsd.org > > X-ASK-Info: Message Queued (2017/08/24 18:09:30) > > X-ASK-Info: Confirmed by User (2017/08/24 18:40:26) > > > > On Fri, Aug 25, 2017 at 12:00 AM, Kirk McKusick <mckusick@mckusick.com> > wrote: > > > >>> To: Aijaz Baig <aijazbaig1@gmail.com>, freebsd-fs@freebsd.org > >>> Subject: Re: Tips on remote debugging for filesystem code > >>> From: Julian Elischer <julian@freebsd.org> > >>> Date: Thu, 24 Aug 2017 16:24:50 +0800 > >>> > >>> On 24/8/17 1:40 pm, Aijaz Baig wrote: > >>> > >>>> ... > >>>> How do you guys normally do it? Keen to hear tips and best practices > >>> > >>> I have had more success recently using BHype to make a Virtual FreeBSD > >>> machine and connecting to it using the built-in gdb interface. > >>> > >>> Firstly it is easier than a serial interface and secondly you don't > >>> need two machines. > >> > >> Is there documentation that describes how to do this? I have been using > >> a patched up set of scripts provided by John Baldwin a couple of years > >> ago to use kgdb with a bhyve VM. But if there is an existing way to do > >> it now, I would rather switch to it. > >> > >> Kirk McKusick > > > > Yes please I would love to hear about this as well. Nonetheless if > > any one of you Kirk or Julian could help me get started a bit with > > what how you guys use it (and those scripts that Kirk talks about), > > it would be really nice. > > Below I enclose an email from John Baldwin that got me going on > debugging under bhyve. I am copying John as he likely has progressed > considerably in the last two years, so can likely provide some more > up to date tips. Though as far as I can see, he never added his > updates to the bhyve/vmrun.sh script. So I am not sure how Julian > has set up his bhyve debugging which is what lead to my initial question. > > > By the way is this bhyve the only way to go for fs code debugging? > > > > Best Regards, > > Aijaz Baig > > You could use multiple machines or run under other hypervisors, but > bhyve lets you do everything on one machine and has the benefit of > easily integrating with kgdb. > > Another option is to read the several chapters in my book that describes > how the VFS layer works ;-) > > Kirk McKusick > > =-=-= > > From: John Baldwin <jhb@freebsd.org> > Date: Tue, 08 Sep 2015 16:53:26 -0700 > To: Kirk McKusick <mckusick@mckusick.com> > Subject: Re: kgdb ported to devel/gdb > > On Tuesday, September 08, 2015 04:05:06 PM Kirk McKusick wrote: > >> From: John Baldwin <jhb@freebsd.org> > >> To: freebsd-current@freebsd.org > >> Subject: kgdb ported to devel/gdb > >> Date: Mon, 31 Aug 2015 14:32:04 -0700 > >> > >> Over the past several months I have ported kgdb to the version of > >> gdb in ports. I have a pending patch to the gdb port to add fork > >> following, but once that is done (and possibly after updating to > >> 7.10) I will try to add my existing work as a KGDB option on the > >> port. Until such time, you can try the newer kgdb by checking out > >> my branch from git. > >> > >> Here's my cheat sheet on how to build the newer kgdb. Note that > >> if you build a world with my cross-libkvm patches you should get a > >> kgdb that can debug i386 cores on amd64 and vice versa. > >> > >> All of the targets that the native devel/gdb support have their > >> backends ported (so x86, sparc64, powerpc and powerpc64). I have > >> not yet ported arm or mips since those don't work for userland yet > >> in upstream gdb. I have only compiled non-x86 backends. Testing > >> of the new kgdb on sparc64 and powerpc would be appreciated. > >> > >> Steps: > >> > >> % git clone https://github.com/bsdjhb/gdb.git > >> % git checkout freebsd-7.9.1-kgdb > >> % fetch http://www.freebsd.org/~jhb/gdb/build > >> % pkg install devel/gdb > >> > >> # Having gdb installed will mean you get the python bindings in the > right > >> # place. > >> > >> % pkg install gmake > >> > >> # I think this is the only build tool you need? > >> > >> % ./build > >> % cd obj > >> > >> # Replace 'obj' with 'obj.<arch>' for all but amd64 > >> > >> % gmake > >> > >> # ... wait > >> > >> You will now have a binary at 'obj/gdb/kgdb'. I just run it from my obj > >> tree currently when testing. Once it becomes part of the port it will > get > >> installed as /usr/local/bin/kgdb791 or some such. > > > > This is very helpful, thanks for doing it. > > > > I recently dumped my (very ancient) i386 debugging machine and started > > debugging in a bhyve virtual machine. Much quicker to boot and test. > > But, I lost the ability to remote debug over a serial line. Is there > > some way to remote attach to my virtual machine so I can once again > > set breakpoints, single step it, etc? > > Yes, I do this with my VMs (and this is how I tested the i386 <--> amd64 > cross debugging). The way I do this is to attach a nmdm device to COM2 > (uart1) in my VMs and then run gdb over that. First, I use a patched > vmrun.sh > to always create a /dev/nmdm<vmname>2{A,B} pair of devices attached to COM2 > on each VM. The 'B' device is used by bhyve, and I use the 'A' device on > the > host with 'target remote' in kgdb. My change to vmrun.sh is this: > > --- /usr/share/examples/bhyve/vmrun.sh 2015-08-15 16:31:57.891092000 > -0700 > +++ bhyve/vmrun.sh 2015-09-08 16:46:05.000000000 -0700 > @@ -91,9 +91,13 @@ > loader_opt="" > bhyverun_opt="-H -A -P" > pass_total=0 > +com2="" > > -while getopts ac:C:d:e:g:hH:iI:m:p:t: c ; do > +while getopts 2:ac:C:d:e:g:hH:iI:m:p:t: c ; do > case $c in > + 2) > + com2=${OPTARG} > + ;; > a) > bhyverun_opt="${bhyverun_opt} -a" > ;; > @@ -269,7 +273,13 @@ > i=$(($i + 1)) > done > > + if [ -n "${com2}" ]; then > + devargs="$devargs -l com2,${com2}" > + elif kldstat -qm nmdm; then > + devargs="$devargs -l com2,/dev/nmdm${vmname}2B" > + fi > + > ${FBSDRUN} -c ${cpus} -m ${memsize} ${bhyverun_opt} \ > -g ${gdbport} \ > -s 0:0,hostbridge \ > -s 1:0,lpc \ > > Note that if you have nmdm kldloaded, this version will place a nmdm > device on COM2 automatically. > > Next, I enable gdb on COM2 by adding 'hint.uart.1.flags="0x80"' to > /boot/device.hints inside the VM. > > To test it, I boot the VM and then run 'sysctl debug.kdb.enter=1' on the > console (or over an ssh session). On the host I run my new kgdb (since > it can do i386 cross debugging via remote out of the box). Note that > you will need the kernel.debug on the host. I do this by keeping my > work tree on my host machine and exporting it via NFS into the guest. > I then build the kernel the "old" way using config and make directly so > that the kernel.debug is at 'sys/i386/compile/<FOO>/kernel.debug'. You > can then run the new kgdb as 'kgdb /path/to/kernel.debug'. In the VM > console, run 'gdb' at the db> prompt to switch over to GDB. You can then > use 'target remote /dev/nmdm<vmname>2A' to connect to it. For example, > if your VM is named vm0, you would use /dev/nmdmvm02A. > > -- > John Baldwin > -- Best Regards, Aijaz Baig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHB2L%2BcrbMGvwAB7zx%2BnaLC5DTbOy7RWbxyFyPCGAZmaLoQ5fg>