From owner-freebsd-hackers@freebsd.org Sat Jul 1 05:16:15 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82B85D8B0C3 for ; Sat, 1 Jul 2017 05:16:15 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 6887F7873D for ; Sat, 1 Jul 2017 05:16:15 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: by mailman.ysv.freebsd.org (Postfix) id 67A61D8B0C0; Sat, 1 Jul 2017 05:16:15 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 672E2D8B0BF for ; Sat, 1 Jul 2017 05:16:15 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-15.reflexion.net [208.70.210.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 271F97873C for ; Sat, 1 Jul 2017 05:16:14 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 15928 invoked from network); 1 Jul 2017 05:20:28 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 1 Jul 2017 05:20:28 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.1) with SMTP; Sat, 01 Jul 2017 01:16:13 -0400 (EDT) Received: (qmail 29572 invoked from network); 1 Jul 2017 05:16:13 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 1 Jul 2017 05:16:13 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 94B7BEC81E6; Fri, 30 Jun 2017 22:16:12 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: extract panic message & debugging from vmcore.0 ? From: Mark Millard In-Reply-To: Date: Fri, 30 Jun 2017 22:16:11 -0700 Cc: hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20170630021205.GA43579@mail.michaelwlucas.com> To: "Michael W. Lucas" X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jul 2017 05:16:15 -0000 [Some more notes prompted by your reply.] On 2017-Jun-29, at 7:35 PM, Mark Millard wrote: > On 2017-Jun-29, at 7:12 PM, Michael W. Lucas wrote: >=20 >>=20 >> # cd /usr/obj/usr/src/sys/GENERIC >> # make gdbinit >> # gdb kernel.debug >=20 > Did you miss a /var/crash/vmcore.0 above? I should have noted that gdbinit is likely for setting up to use gdb remotely against a live system for remote kernel debugging. (Not something that I've ever done.) It likely does nothing useful for vmcore.* inspection. gdb itself does not deal with vmcore.* files to my knowledge. So this likely was instead a false direction for your purposes. > Separately. . . >=20 > I'm no expert but you might need to pick > between the default minidump vs. a full > memory dump depending on purposes and what > information is to be extracted: >=20 > debug.minidump: 1 > vs. > debug.minidump: 0 >=20 > in the sysctl debug.minidump output. I > doubt that they are fully equivalent. >=20 > Of course a full memory dump implies > needing space ready to hold that full > copy. I'll also note that virtual addresses vs. physical addresses are likely an issue for at least debug.minidump=3D0 . As for the crashinfo that you mention in your reply: On head -320482 . . . # grep -r crashinfo /etc /etc/rc.d/savecore: if checkyesno crashinfo_enable; then /etc/rc.d/savecore: ${crashinfo_program} -b -d = ${dumpdir} /etc/defaults/rc.conf:crashinfo_enable=3D"YES" # Automatically generate = crash dump summary. /etc/defaults/rc.conf:crashinfo_program=3D"/usr/sbin/crashinfo" # Script = to generate crash dump summary. Normally the boot sequence produces a crashinfo report after creating the vmcore.* file, at least if crashinfo is enabled. The context for the below is a powerpc with debug.minidump=3D0 (disabled) and a production kernel for the failures. (The debug kernel does not get the problems involved for head -r320482 .) vmcore.*'s, core.txt.*'s, info.*'s and *.last's were all produced towards the end of the boot sequence before the login prompt each time. # ls -lT /var/crash/ total 2685292 -rw-r--r-- 1 root wheel 2 Jun 29 23:17:52 2017 bounds -rw------- 1 root wheel 0 Jun 29 22:47:14 2017 core.txt.5 -rw------- 1 root wheel 26573 Jun 29 23:18:51 2017 core.txt.6 -rw------- 1 root wheel 351 Jun 29 22:46:15 2017 info.5 -rw------- 1 root wheel 378 Jun 29 23:17:52 2017 info.6 lrwxr-xr-x 1 root wheel 0 Jun 29 22:47:06 2017 info.last ->=20= -rw-r--r-- 1 root wheel 5 Feb 22 02:37:33 2016 minfree -rw------- 1 root wheel 2147487744 Jun 29 22:47:06 2017 vmcore.5 -rw------- 1 root wheel 2147487744 Jun 29 23:18:43 2017 vmcore.6 lrwxr-xr-x 1 root wheel 0 Jun 29 22:47:06 2017 vmcore.last = ->=20 (There are oddities for the system that have made the links and core.txt.5 empty.) # more /var/crash/info.5 Dump header from device: /dev/label/FBSDG4Sswap Architecture: powerpc Architecture Version: 1 Dump Length: 2147487744 Blocksize: 512 Dumptime: Thu Jun 29 22:34:22 2017 Hostname: FBSDG4S Magic: FreeBSD Kernel Dump Version String: FreeBSD 12.0-CURRENT r320482M Panic String:=20 Dump Parity: 4013015019 Bounds: 5 Dump Status: good core.txt.6 attempted to have sections for: ps -axlww vmstat -s vmstat -m vmstat -z vmstat -i pstat -T pstat -s iostat ipcs -a ipcs -T nfsstat netstat -s netstat -m netstat -anA netstat -aL fstat dmesg kernel config ddb capture buffer (That does not mean that they all were useful/accurate.) I'll not list the text here. Notes: powerpc is not actually supported for kgdb and the like and I had to change some things to get this much working as well as it does for me now. (It is limited but has been useful to me.) =3D=3D=3D Mark Millard markmi at dsl-only.net