From owner-freebsd-hackers@freebsd.org Mon Feb 27 05:35:14 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06070CEF5D2 for ; Mon, 27 Feb 2017 05:35:14 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C1EB07E4 for ; Mon, 27 Feb 2017 05:35:13 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: by mail-oi0-x22d.google.com with SMTP id f192so14477278oic.3 for ; Sun, 26 Feb 2017 21:35:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=yowbqyMHSMPzFEMw/DzFfy0z7OA41R9VRXYTY4Lb+B8=; b=TLRdVJB2xs8OZxkO2vgQVZwbcGVykHrnF3y/Zu6UDlO6j9ESay0Q749o3Ddp1PZKc3 H0wcUmgGzU9Q+iDMgpd60qEhuhwtERooFuBh9C1FGLdvDHIT+SS+7txESTzswYEqb/vu e3mmsLB1hAHQQtiWGqlDtpLzQzBfKzI/2k2yf0bCeLK/hmeCCWnOl5r9GZN5PoYSocl5 2OGdpFUJon6qqijNn582ti6+ypdGP/yzq3lBZ2jm5Cl/BomUfCwdpdK684vkuh5GP67V hrHSJFK/Um0i649gTQL5SyDtZXwASogVsIpUpvFUqz429kVwNMkdxowntey7E9uG9Pm3 lx7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=yowbqyMHSMPzFEMw/DzFfy0z7OA41R9VRXYTY4Lb+B8=; b=SmKXOyPjvkNhLo4YsCWolTsVK/lXcpoptUGfD5nKoh42cRTmxpo+oPm+U4ZGsenroC FBRMHLxIcKa4lVQgOyAgfYCvhihO4K19h/RA01ervSxMjKYI0e0ewtmEASWj2tWoHm3I dxZrbqps/KOGLdq9LtCHv7bzbZgx/osoOeOtsTF/DYMYzKmbVRjJs3qrZjidzl2las/z FRQEW9D6jzkLPVhN3c3Q6kwenjCUHXTbssny9m7b5H9DKtX27gD5y9k0rhDYRWpmx3LS H8I8JLVcSXp/yCNVzY2LRb6t8sMZXMUkbIDLsicxSEg0b6ZwJnXTZshALOj83v816Vul DscQ== X-Gm-Message-State: AMke39nKkgA8lnTi1pcd/7hEjvQEQSsufN48Wa1jVELVVaJ29epLKs3yYyNlw1N845B6JxDtjd93vgTa7aRtMg== X-Received: by 10.202.85.193 with SMTP id j184mr7045966oib.199.1488173712879; Sun, 26 Feb 2017 21:35:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.54.130 with HTTP; Sun, 26 Feb 2017 21:34:32 -0800 (PST) From: Lee D Date: Mon, 27 Feb 2017 00:34:32 -0500 Message-ID: Subject: How approach debugging a kernel crash? To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Feb 2017 05:35:14 -0000 Hi, I am trying to write a custom boot loader for ARM, to replace u-boot and ubldr. As I'm working through this, I keep getting kernel crashes. I've got the kernel debugger enabled, but doing a backtrace doesn't reveal any useful information. How does one go about figuring out exactly what caused an exception? I need to know where the kernel crashed so I can figure out what piece of hardware I haven't set up correctly. The back trace is just a bunch of abort stuff, and ends in the message "Unable to unwind into user mode". I've quoted the backtrace below, and also my kernel message. Mostly I'm looking for suggestions on how to go about finding the location of the crash, as I expect to be doing this a lot this week :-) Thanks! Lee db> bt Tracing pid 0 tid 100000 td 0xc08f8470 db_trace_self() at db_trace_self pc = 0xc0669b44 lr = 0xc014c288 (db_hex2dec+0x1f4) sp = 0xffff0cb0 fp = 0xffff0cc8 db_hex2dec() at db_hex2dec+0x1f4 pc = 0xc014c288 lr = 0xc014becc (db_command_loop+0x2f4) sp = 0xffff0cd0 fp = 0xffff0d70 r4 = 0x00000001 r5 = 0x00000000 r6 = 0xc0704ae6 r10 = 0xc08f6f98 db_command_loop() at db_command_loop+0x2f4 pc = 0xc014becc lr = 0xc014bc4c (db_command_loop+0x74) sp = 0xffff0d78 fp = 0xffff0d88 r4 = 0xc06cfe7d r5 = 0xc06e1e0e r6 = 0xc08f6f84 r7 = 0xffff0fa0 r8 = 0xc08ead98 r9 = 0xc0791060 r10 = 0xc08ead9c db_command_loop() at db_command_loop+0x74 pc = 0xc014bc4c lr = 0xc014f084 (db_fetch_ksymtab+0x2e8) sp = 0xffff0d90 fp = 0xffff0ea8 r4 = 0x00000807 r5 = 0x00000000 r6 = 0xc08f6f90 r10 = 0xc08ead9c db_fetch_ksymtab() at db_fetch_ksymtab+0x2e8 pc = 0xc014f084 lr = 0xc0341870 (kdb_trap+0x180) sp = 0xffff0eb0 fp = 0xffff0ed8 r4 = 0x00000000 r5 = 0x00000807 r6 = 0xc08eadb8 r10 = 0xc08ead9c kdb_trap() at kdb_trap+0x180 pc = 0xc0341870 lr = 0xc06908b4 (abort_handler+0x678) sp = 0xffff0ee0 fp = 0xffff0f00 r4 = 0xffff0fa0 r5 = 0x00000013 r6 = 0xffff1030 r7 = 0x00000007 r8 = 0x00000807 r9 = 0xc08f8470 r10 = 0xffff0fa0 abort_handler() at abort_handler+0x678 pc = 0xc06908b4 lr = 0xc0690600 (abort_handler+0x3c4) sp = 0xffff0f08 fp = 0xffff0f98 r4 = 0x00000001 r5 = 0x00000007 r6 = 0x00000000 r7 = 0x00000807 r8 = 0x00000013 r10 = 0xffff0fa0 abort_handler() at abort_handler+0x3c4 pc = 0xc0690600 lr = 0xc066c42c (exception_exit) sp = 0xffff0fa0 fp = 0xc0a13e70 r4 = 0x00000000 r5 = 0xc08f8808 r6 = 0x00000001 r7 = 0x00000000 r8 = 0xc08f890c r9 = 0xc08f8908 r10 = 0x00002802 exception_exit() at exception_exit pc = 0xc066c42c lr = 0x1000019c (0x1000019c) sp = 0xffff1034 fp = 0xc0a13e70 r0 = 0xc066c534 r1 = 0xc0a0b000 r2 = 0xffff107c r3 = 0x20010193 r4 = 0x00000000 r5 = 0xc08f8808 r6 = 0x00000001 r7 = 0x00000000 r8 = 0xc08f890c r9 = 0xc08f8908 r10 = 0x00002802 r12 = 0xfefefeff data_abort_entry() at data_abort_entry+0x30 pc = 0xc066c534 lr = 0x1000019c (0x1000019c) sp = 0xffff1034 fp = 0xc0a13e70 Unable to unwind into user mode KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-RELEASE-p1 #27 r309723M: Sat Feb 25 18:51:15 EST 2017 builder@abe:/usr/home/builder/projects/fbsd_11.0.1/obj/arm.armv6/usr/home/builder/projects/fbsd_11.0.1/src/sys/AXSACM arm FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) VT: init without driver. CPU: Cortex A9-r3 rev 0 (Cortex-A core) Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext WB enabled LABT branch prediction disabled LoUU:2 LoC:2 LoUIS:2 Cache level 1: 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 4-way instruction cache Read-Alloc real memory = 535822336 (511 MB) avail memory = 513486848 (489 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs random: entropy device external interface ofwbus0: simplebus0: on ofwbus0 simplebus1: on ofwbus0 simplebus2: on ofwbus0 l2cache0: mem 0xf02000-0xf02fff on simplebus0 l2cache0: cannot allocate IRQ, not using interrupt l2cache0: Part number: 0x3, release: 0x8 l2cache0: L2 Cache enabled: 512KB/32B 8 ways gic0: mem 0xf01000-0xf01fff,0xf00100-0xf001ff on simplebus0 gic0: pn 0x390, arch 0x1, rev 0x2, implementer 0x43b irqs 96 mp_tmr0: mem 0xf00200-0xf002ff,0xf00600-0xf0061f on simplebus0 Timecounter "MPCore" frequency 325000000 Hz quality 800 Event timer "MPCore" frequency 325000000 Hz quality 1000 zy7_slcr0: mem 0-0xfff on simplebus0 zy7_devcfg0: mem 0x7000-0x7fff on simplebus0 uart0: mem 0x1000-0x1fff on simplebus1 uart0: console (-1,n,8,1) ehci0: mem 0x2000-0x2fff on simplebus1 usbus0: EHCI version 1.0 usbus0: stop timeout usbus0 on ehci0 gpio0: mem 0xa000-0xafff on simplebus1 gpiobus0: on gpio0 gpioc0: on gpio0 cgem0: mem 0xb000-0xbfff on simplebus1 miibus0: on cgem0 rgephy0: PHY 0 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto rgephy1: PHY 1 on miibus0 rgephy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto cgem0: no mac address found, assigning random: 62:73:64:b9:65:d2 cgem0: Ethernet address: 62:73:64:b9:65:d2 sdhci_fdt0: mem 0x100000-0x100fff on simplebus1 sdhci_fdt0: 1 slot(s) allocated mmc0: on sdhci_fdt0 sdhci_fdt1: mem 0x101000-0x101fff on simplebus1 sdhci_fdt1: 1 slot(s) allocated mmc1: on sdhci_fdt1 cryptosoft0: Fatal kernel mode data abort: 'Translation Fault (L2)' on write trapframe: 0xffff0fa0 FSR=00000807, FAR=ffff1030, spsr=20010193 r0 =c066c534, r1 =c0a0b000, r2 =ffff107c, r3 =20010193 r4 =00000000, r5 =c08f8808, r6 =00000001, r7 =00000000 r8 =c08f890c, r9 =c08f8908, r10=00002802, r11=c0a13e70 r12=fefefeff, ssp=ffff1034, slr=1000019c, pc =c066c534 [ thread pid 0 tid 100000 ] Stopped at data_abort_entry+0x30: str r0, [r13, -#0x004]!