From owner-svn-doc-all@freebsd.org Wed Apr 18 23:48:43 2018 Return-Path: Delivered-To: svn-doc-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A0BBF8D03D; Wed, 18 Apr 2018 23:48:43 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA740808E2; Wed, 18 Apr 2018 23:48:42 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A0FD821C0F; Wed, 18 Apr 2018 23:48:42 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w3INmgVT087315; Wed, 18 Apr 2018 23:48:42 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w3INmg4U087314; Wed, 18 Apr 2018 23:48:42 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201804182348.w3INmg4U087314@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Wed, 18 Apr 2018 23:48:42 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r51572 - head/en_US.ISO8859-1/books/developers-handbook/kerneldebug X-SVN-Group: doc-head X-SVN-Commit-Author: jhb X-SVN-Commit-Paths: head/en_US.ISO8859-1/books/developers-handbook/kerneldebug X-SVN-Commit-Revision: 51572 X-SVN-Commit-Repository: doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2018 23:48:43 -0000 Author: jhb Date: Wed Apr 18 23:48:42 2018 New Revision: 51572 URL: https://svnweb.freebsd.org/changeset/doc/51572 Log: Various updates to the kernel debugging chapter. - Document vmcore.last and describe it as the way to find the most recent dump rather than the highest numbered dump. - Document crashinfo and that it automatically runs to generate a core.txt.N file if core dumps are enabled in rc.conf. - Add a section on testing kernel dumps via the debug.kdb.panic sysctl. Remove a later note about debug.kdb.panic from the DDB section. - Remove any mention of gdb -k (for pre 5-3 kernels) and just talk about kgdb. - Remove paragraph that talks about trying to find the kernel.debug file. Instead, recommand 'kgdb -n ' which does this lookup automatically, and specifically recommend 'kgb -n last' to open the most recent crash dump. Mention the fallback of specifying the kernel and vmcore directly if needed. - Remove example dump from FreeBSD 2. It is generally no longer relevant. It used gdb -k which uses a different stack trace format as well as including a 'frame' command that doesn't existing kgdb. (kgdb instead lets you switch to different threads and processes). - Remove mention of old boot blocks that don't load debug symbols. I think this was last relevant in FreeBSD 2.x or 3.x. - Rework the description of 'boot -d' to assume the boot menu and explicitly mention 'boot -d' at the loader prompt. - Document how to get stack traces of other threads in DDB. - Fix a few references to gdb to reference kgdb instead. - Replace 'call cpu_reset' with 'reset' for DDB. Differential Revision: https://reviews.freebsd.org/D14711 Modified: head/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.xml Modified: head/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.xml ============================================================================== --- head/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.xml Wed Apr 18 19:25:52 2018 (r51571) +++ head/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.xml Wed Apr 18 23:48:42 2018 (r51572) @@ -136,12 +136,20 @@ dumpdir is set to), the kernel will increment the trailing number for every crash to avoid overwriting an existing vmcore (e.g., - vmcore.1). While debugging, it is - highly likely that you will want to use the highest version - vmcore in - /var/crash when searching for the right - vmcore. + vmcore.1). &man.savecore.8; will always + create a symbolic link to named vmcore.last + in /var/crash after a dump is saved. + This symbolic link can be used to locate the name of the most + recent dump. + The &man.crashinfo.8; utility generates a text file + containing a summary of information from a full memory dump + or minidump. If dumpdev has been set in + &man.rc.conf.5;, &man.crashinfo.8; will be invoked + automatically after &man.savecore.8;. The output is saved + to a file in dumpdir named + core.txt.N. + If you are testing a new kernel but need to boot a different one in order to get your system up and running again, boot it only into single @@ -161,45 +169,61 @@ device as it is likely different than /dev/ad0s1b! + + + Testing Kernel Dump Configuration + + The kernel includes a &man.sysctl.8; node that requests a + kernel panic. This can be used to verify that your system is + properly configured to save kernel crash dumps. You may wish + to remount existing file systems as read-only in single user + mode before triggering the crash to avoid data loss. + + &prompt.root; shutdown now +... +Enter full pathname of shell or RETURN for /bin/sh: +&prompt.root; mount -a -u -r +&prompt.root; sysctl debug.kdb.panic=1 +debug.kdb.panic:panic: kdb_sysctl_panic +... + + After rebooting, your system should save a dump in + /var/crash along with a matching summary + from &man.crashinfo.8;. + Debugging a Kernel Crash Dump with <command>kgdb</command> - This section covers &man.kgdb.1; as found in &os; 5.3 - and later. In previous versions, one must use - gdb -k to read a core dump file. - Since &os; 12 kgdb is acquired by installing - devel/gdb. + This section covers &man.kgdb.1;. The latest version is + included in the devel/gdb. An older version + is also present in &os; 11 and earlier. - Once a dump has been obtained, getting useful information - out of the dump is relatively easy for simple problems. Before - launching into the internals of &man.kgdb.1; to debug - the crash dump, locate the debug version of your kernel - (normally called kernel.debug) and the path - to the source files used to build your kernel (normally - /usr/obj/usr/src/sys/KERNCONF - or - /usr/obj/usr/src/amd64.amd64/sys/KERNCONF, - where amd64.amd64 - is the architecture and - KERNCONF - is the ident specified in a kernel - &man.config.5;). With those two pieces of info, let the - debugging commence! - To enter into the debugger and begin getting information - from the dump, the following steps are required at a minimum: + from the dump, start kgdb: - &prompt.root; cd /usr/obj/usr/src/sys/KERNCONF -&prompt.root; kgdb kernel.debug /var/crash/vmcore.0 + &prompt.root; kgdb -n N + Where N is the suffix of the + vmcore.N to + examine. To open the most recent dump use: + + &prompt.root; kgdb -n last + + Normally, &man.kgdb.1; should be able to locate the kernel + running at the time the dump was generated. If it is not able to + locate the correct kernel, pass the pathname of the kernel and + dump as two arguments to kgdb: + + &prompt.root; kgdb /boot/kernel/kernel /var/crash/vmcore.0 + You can debug the crash dump using the kernel sources just like you can for any other program. - This first dump is from a 5.2-BETA kernel and the crash + This dump is from a 5.2-BETA kernel and the crash comes from deep within the kernel. The output below has been modified to include line numbers on the left. This first trace inspects the instruction pointer and obtains a back trace. The @@ -301,173 +325,12 @@ 88:#20 0xc070ca4d in Xint0x80_syscall () at {standard input}:136 89:---Can't read userspace from dump, or kernel process--- 90:(kgdb) quit - - - This next trace is an older dump from the FreeBSD 2 time - frame, but is more involved and demonstrates more of the - features of gdb. Long lines have been folded - to improve readability, and the lines are numbered for - reference. Despite this, it is a real-world error trace taken - during the development of the pcvt console driver. - - 1:Script started on Fri Dec 30 23:15:22 1994 - 2:&prompt.root; cd /sys/compile/URIAH - 3:&prompt.root; gdb -k kernel /var/crash/vmcore.1 - 4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel -...done. - 5:IdlePTD 1f3000 - 6:panic: because you said to! - 7:current pcb at 1e3f70 - 8:Reading in symbols for ../../i386/i386/machdep.c...done. - 9:(kgdb) backtrace -10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767) -11:#1 0xf0115159 in panic () -12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698) -13:#3 0xf010185e in db_fncall () -14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073) -15:#5 0xf0101711 in db_command_loop () -16:#6 0xf01040a0 in db_trap () -17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723) -18:#8 0xf019d2eb in trap_fatal (...) -19:#9 0xf019ce60 in trap_pfault (...) -20:#10 0xf019cb2f in trap (...) -21:#11 0xf01932a1 in exception:calltrap () -22:#12 0xf0191503 in cnopen (...) -23:#13 0xf0132c34 in spec_open () -24:#14 0xf012d014 in vn_open () -25:#15 0xf012a183 in open () -26:#16 0xf019d4eb in syscall (...) -27:(kgdb) up 10 -28:Reading in symbols for ../../i386/i386/trap.c...done. -29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\ -30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\ -31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\ -32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\ -33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\ -34:ss = -266427884}) (../../i386/i386/trap.c line 283) -35:283 (void) trap_pfault(&frame, FALSE); -36:(kgdb) frame frame->tf_ebp frame->tf_eip -37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done. -38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\ -39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403) -40:403 return ((*linesw[tp->t_line].l_open)(dev, tp)); -41:(kgdb) list -42:398 -43:399 tp->t_state |= TS_CARR_ON; -44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */ -45:401 -46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200) -47:403 return ((*linesw[tp->t_line].l_open)(dev, tp)); -48:404 #else -49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag)); -50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */ -51:407 } -52:(kgdb) print tp -53:Reading in symbols for ../../i386/i386/cons.c...done. -54:$1 = (struct tty *) 0x1bae -55:(kgdb) print tp->t_line -56:$2 = 1767990816 -57:(kgdb) up -58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\ -59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126) -60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p)); -61:(kgdb) up -62:#2 0xf0132c34 in spec_open () -63:(kgdb) up -64:#3 0xf012d014 in vn_open () -65:(kgdb) up -66:#4 0xf012a183 in open () -67:(kgdb) up -68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\ -69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\ -70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \ -71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \ -72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673) -73:673 error = (*callp->sy_call)(p, args, rval); -74:(kgdb) up -75:Initial frame selected; you cannot go up. -76:(kgdb) quit - Comments to the above script: - - - - line 6: - - - This is a dump taken from within DDB (see below), hence the - panic comment because you said to!, and a rather - long stack trace; the initial reason for going into DDB has been a - page fault trap though. - - - - - line 20: - - - This is the location of function trap() - in the stack trace. - - - - - line 36: - - - Force usage of a new stack frame; this is no longer necessary. - The stack frames are supposed to point to the right - locations now, even in case of a trap. - From looking at the code in source line 403, there is a - high probability that either the pointer access for - tp was messed up, or the array access was out of - bounds. - - - - - line 52: - - - The pointer looks suspicious, but happens to be a valid - address. - - - - - line 56: - - - However, it obviously points to garbage, so we have found our - error! (For those unfamiliar with that particular piece of code: - tp->t_line refers to the line discipline of - the console device here, which must be a rather small integer - number.) - - - - If your system is crashing regularly and you are running out of disk space, deleting old vmcore files in /var/crash could save a considerable amount of disk space! - - Debugging a Crash Dump with DDD - - Examining a kernel crash dump with a graphical debugger like - ddd is also possible (you will need to install - the devel/ddd port in order to use the - ddd debugger). Add the - option to the ddd command line you would use - normally. For example; - - &prompt.root; ddd --debugger kgdb kernel.debug /var/crash/vmcore.0 - - You should then be able to go about looking at the crash dump using - ddd's graphical interface. - - On-Line Kernel Debugging Using DDB @@ -481,7 +344,7 @@ breakpoints, single-stepping kernel functions, examining and changing kernel variables, etc. However, it cannot access kernel source files, and only has access to the global and static symbols, not to the full - debug information like gdb does. + debug information like kgdb does. To configure your kernel to include DDB, add the options @@ -491,19 +354,13 @@ to your config file, and rebuild. (See The FreeBSD Handbook for details on configuring the FreeBSD kernel). - - If you have an older version of the boot blocks, your - debugger symbols might not be loaded at all. Update the boot blocks; - the recent ones load the DDB symbols automatically. - - Once your DDB kernel is running, there are several ways to enter - DDB. The first, and earliest way is to type the boot flag - right at the boot prompt. The kernel will start up + DDB. The first, and earliest way is to use the boot flag + . The kernel will start up in debug mode and enter DDB prior to any device probing. Hence you can - even debug the device probe/attach functions. Users of &os.current; - will need to use the boot menu option, six, to escape to a command - prompt. + even debug the device probe/attach functions. To use this, exit + the loader's boot menu and enter boot -d at + the loader prompt. The second scenario is to drop to the debugger once the system has booted. There are two simple ways to accomplish @@ -511,10 +368,6 @@ command prompt, simply type the command: &prompt.root; sysctl debug.kdb.enter=1 - - To force a panic on the fly, issue the following command: - &prompt.root; sysctl debug.kdb.panic=1 - Alternatively, if you are at the system console, you may use a hot-key on the keyboard. The default break-to-debugger @@ -556,15 +409,13 @@ continue - To get a stack trace, use: + To get a stack trace of the current thread, use: trace - - Note that when entering DDB via a hot-key, the kernel is currently - servicing an interrupt, so the stack trace might be not of much use - to you. - + To get a stack trace of an arbitrary thread, specify a + process ID or thread ID as a second argument to + trace. If you want to remove a breakpoint, use @@ -662,10 +513,7 @@ panic This will cause your kernel to dump core and reboot, so you can - later analyze the core on a higher level with gdb. - This command - usually must be followed by another continue - statement. + later analyze the core on a higher level with &man.kgdb.1;. call boot(0) @@ -675,7 +523,7 @@ the disk and filesystem interfaces of the kernel are not damaged, this could be a good way for an almost clean shutdown. - call cpu_reset() + reset This is the final way out of disaster and almost the same as hitting the Big Red Button.