From owner-freebsd-current@freebsd.org Wed Jul 8 01:06:24 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2814093CC for ; Wed, 8 Jul 2015 01:06:24 +0000 (UTC) (envelope-from jason.unovitch@gmail.com) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D3BD51DFF for ; Wed, 8 Jul 2015 01:06:23 +0000 (UTC) (envelope-from jason.unovitch@gmail.com) Received: by ykeo3 with SMTP id o3so73645207yke.0 for ; Tue, 07 Jul 2015 18:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=z65iLJJR+e2xBuz2+b5I6m/pYFvcsFxFa1j7SXW13zc=; b=d86ECxtqQ0p6/aF5kLKCGYsEnZAyt4OQ9cWOYHqhEIU0BupI2kebLq6NYqZgTrvGmP eO/h1oMh4ePxEsMCt9ulSZwXU1RKdmKWmzj2+BZwgIboscb/0BXdFhYPlVR2Tb4xEh6x gHus3ZsbW6Hh6djxazpm9JIxl11XY7b5O+y8VguT1GAh+AIPGq8iFi8rvXI+NPW4iH9b zMt0ZaTBkPiQA8GOsEmIcERAGV/5TiuMZB9IgOSQtELzIFbjxCpszz418j5aYAFwoBHn AW0ueLfRjO4cJtlR76ElSZUtVr5zZAC50bHC1qlk1sRaM4GEkQa4uadD3a0mTh+WqzdV 6fgA== X-Received: by 10.13.204.142 with SMTP id o136mr8462774ywd.171.1436317582938; Tue, 07 Jul 2015 18:06:22 -0700 (PDT) Received: from ?IPv6:2606:a000:568d:3602:be5f:f4ff:fe5d:f28? ([2606:a000:568d:3602:be5f:f4ff:fe5d:f28]) by smtp.googlemail.com with ESMTPSA id x8sm571539ywa.41.2015.07.07.18.06.22 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Jul 2015 18:06:22 -0700 (PDT) Message-ID: <559C778B.1080209@gmail.com> Date: Tue, 07 Jul 2015 21:06:19 -0400 From: Jason Unovitch User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org Subject: Boot time panic on recent i386 build after SMP: AP CPU #1 Launched Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2015 01:06:24 -0000 Evening, I have a Soekris 6501 running i386 11-CURRENT that I had seen panic at boot when updating this weekend. It'll panic right after the "SMP: AP CPU #1 Launched" and go into a long loop of __mtx_lock_spin_flags, i8254_get_timecount, binuptime, _mtx_lock_spin_cookie. An abbreviated output is at the bottom of the email while the full output is linked below. - My currently working build's uname -a: FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r284194: Tue Jun 9 23:38:40 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/sys/IPSEC i386 - The misbehaving builds: FreeBSD 11.0-CURRENT #0 r285155: Sun Jul 5 07:35:36 UTC 2015 root@xts-bsd.:/usr/obj/nanobsd.soekris/i386.i386/usr/src/sys/GENERIC i386 - Full boot log: http://pastebin.com/QmzpnZdf FreeBSD 11.0-CURRENT #0 r285179: Mon Jul 6 00:10:45 UTC 2015 root@xts-bsd.:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/GENERIC-NODEBUG i386 - Full boot log: http://pastebin.com/j9ZRdK3V I tested 10.2-PRERELEASE and didn't see any issues so this appears to be confined to CURRENT for me. I was experimenting with trying out a GENERIC kernel since IPSEC is available there now but otherwise I was just doing a routine update. For the time being I just reverted to the prior NanoBSD image slice. I'm going to start to a binary search to nail down where the issue came about. I'll test each evening after work with a new image but this is my home router/wifi ap so there a minor inconvenience while I do so and it will take a few days. I'll report what I find. Any ideas on where to start would be appreciated! Thanks for all the good work you do! Jason -- Truncated panic output follows (full is at links above): SMP: AP CPU #1 Launched! Timecounter "TSC" frequency 1600030024 Hz quality 1000 Fatal double fault: eip = 0xc0bfe063 esp = 0xe1fbc000 ebp = 0xe1fbc000 cpuid = 1; apic id = 01 panic: double fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper(c137c62d,c1229194,ff,0,c15c1f1c,...) at db_trace_self_wrapper+0x2a/frame 0xc171fc20 kdb_backtrace(c15381f0,1,c152f513,c171fcdc,1,...) at kdb_backtrace+0x2d/frame 0xc171fc88 vpanic(c152f513,c171fcdc,c171fcdc,c171fcf4,c1200186,...) at vpanic+0x117/frame 0xc171fcbc panic(c152f513,1,1,1,e1fbc000,...) at panic+0x1b/frame 0xc171fcd0 dblfault_handler() at dblfault_handler+0xa6/frame 0xc171fcd0 --- trap 0x17, eip = 0xc0bfe063, esp = 0xe1fbc000, ebp = 0xe1fbc000 --- lockstat_nsecs(aa55aa55,aa55aa55,aa55aa55,aa55aa55,aa55aa55,...) at lockstat_nsecs+0x3/frame 0xe1fbc000 _mtx_lock_spin_cookie(c1774d64,c6efc960,0,0,0,...) at _mtx_lock_spin_cookie+0x42/frame 0xe1fbc064 getit(c7163400,aa55aa55,aa55aa55,aa55aa55,aa55aa55,...) at getit+0x9d/frame 0xe1fbc08c i8254_get_timecount(c7119118,9751ad54,44,c6efc960,c6efc960,...) at i8254_get_timecount+0x89/frame 0xe1fbc0b8 binuptime(e1fbc0e8,aa55aa55,b,34d2b2cb,3fd28fa4,...) at binuptime+0x46/frame 0xe1fbc0d8 ... Repeat.... over and over and over with steady increments to memory addresses... lockstat_nsecs(0,13669f,e1fbdb40,c0bcb188,c1757704,...) at lockstat_nsecs+0x16/frame 0xe1fbdb00 thread_lock_flags_(c6efc960,0,c137adf2,87a,e1fadcac,...) at thread_lock_flags_+0x24f/frame 0xe1fbdb68 sched_preempt(c6efc960,0,1,c176cf80,0,...) at sched_preempt+0x65/frame 0xe1fbdb90 ipi_bitmap_handler(8,28,28,0,c176cf80,...) at ipi_bitmap_handler+0x4d/frame 0xe1fbdbb0 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2f/frame 0xe1fbdbb0 --- interrupt, eip = 0xc121a893, esp = 0xe1fbdbf8, ebp = 0xe1fbdc04 --- cpu_idle_acpi(7e8f33c7,0,c16f4780,12,c16f4808,...) at cpu_idle_acpi+0x43/frame 0xe1fbdc04 cpu_idle(0,e1fbdc78,aa55aa55,aa55aa55,aa55aa55,...) at cpu_idle+0x9d/frame 0xe1fbdc20 sched_idletd(0,e1fbdce8,aa55aa55,aa55aa55,aa55aa55,...) at sched_idletd+0x341/frame 0xe1fbdca4 fork_exit(c0c5fa40,0,e1fbdce8) at fork_exit+0xa2/frame 0xe1fbdcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe1fbdcd4 --- trap 0, eip = 0, esp = 0xe1fbdd20, ebp = 0 --- KDB: enter: panic [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why