From owner-freebsd-current@FreeBSD.ORG Mon Apr 29 17:02:46 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DC59BF4A; Mon, 29 Apr 2013 17:02:46 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8051F107E; Mon, 29 Apr 2013 17:02:46 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D13F1B964; Mon, 29 Apr 2013 13:02:45 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Subject: Re: panic: in_pcblookup_local (?) Date: Mon, 29 Apr 2013 12:24:06 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <20130428040256.GK1611@glenbarber.us> In-Reply-To: <20130428040256.GK1611@glenbarber.us> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201304291224.06328.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 29 Apr 2013 13:02:45 -0400 (EDT) Cc: Glen Barber , Ian FREISLICH X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Apr 2013 17:02:46 -0000 On Sunday, April 28, 2013 12:02:56 am Glen Barber wrote: > On Sat, Apr 27, 2013 at 10:17:32AM +0200, Ian FREISLICH wrote: > > Hi > > > > I've been getting the following panic on recent current r249717. > > Sadly the crashdump is useless. > > > > I just saw similar panic on 10-CURRENT r249588. > > > Fatal trap 9: general protection fault while in kernel mode > > cpuid = 15; apic id = 0f > > instruction pointer = 0x20:0xffffffff80546fbc > > stack pointer = 0x28:0xffffff846b677770 > > frame pointer = 0x28:0xffffff846b6777b0 > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 0 > > current process = 4361 (zabbix_agentd) > > Hmm.. This is interests me. In my case, cf-agent was the current > process. > > Backtrace of my panic follows. Any pointers on how to debug this > further would be appreciated. > > Glen > > Script started on Sat Apr 27 23:53:53 2013 > root@orion:/usr/obj/usr/src/sys/ORION # kgdb ./kernel.debug /var/crash/vmcore.4 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > > Fatal trap 9: general protection fault while in kernel mode > cpuid = 1; apic id = 01 > instruction pointer = 0x20:0xffffffff80736cec > stack pointer = 0x28:0xffffff81aad4e760 > frame pointer = 0x28:0xffffff81aad4e7a0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 78664 (cf-agent) > trap number = 9 > panic: general protection fault > cpuid = 1 > KDB: stack backtrace: > #0 0xffffffff80642a56 at kdb_backtrace+0x66 > #1 0xffffffff80606eeb at panic+0x13b > #2 0xffffffff808e3b10 at trap_fatal+0x290 > #3 0xffffffff808e4331 at trap+0x241 > #4 0xffffffff808cdbb3 at calltrap+0x8 > #5 0xffffffff807371d8 at in_pcb_lport+0x128 > #6 0xffffffff8073745a at in_pcbbind_setup+0x16a > #7 0xffffffff80737d8e at in_pcbconnect_setup+0x71e > #8 0xffffffff80737df9 at in_pcbconnect_mbuf+0x59 > #9 0xffffffff807bf29f at udp_connect+0x11f > #10 0xffffffff80680615 at kern_connectat+0x275 > #11 0xffffffff80680731 at sys_connect+0x41 > #12 0xffffffff808e32cb at amd64_syscall+0x63b > #13 0xffffffff808cde97 at Xfast_syscall+0xf7 > Uptime: 3d19h38m52s > (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 > (ada0:ahcich0:0:0:0): CAM status: CCB request is in progress > (ada0:ahcich0:0:0:0): Error 5, Retries exhausted > (ada0:ahcich0:0:0:0): Synchronize cache failed > (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 > (ada1:ahcich1:0:0:0): CAM status: CCB request is in progress > (ada1:ahcich1:0:0:0): Error 5, Retries exhausted > (ada1:ahcich1:0:0:0): Synchronize cache failed > (ada2:ahcich4:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 > (ada2:ahcich4:0:0:0): CAM status: CCB request is in progress > (ada2:ahcich4:0:0:0): Error 5, Retries exhausted > (ada2:ahcich4:0:0:0): Synchronize cache failed > (ada3:ahcich5:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 > (ada3:ahcich5:0:0:0): CAM status: CCB request is in progress > (ada3:ahcich5:0:0:0): Error 5, Retries exhausted > (ada3:ahcich5:0:0:0): Synchronize cache failed > Dumping 1014 out of 6049 MB:..2%..12%..21%..32%..42%..51%..62%..71%..81%..92% > > Reading symbols from /boot/kernel/zfs.ko.symbols...done. > Loaded symbols for /boot/kernel/zfs.ko.symbols > Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. > Loaded symbols for /boot/kernel/opensolaris.ko.symbols > #0 doadump (textdump=) at pcpu.h:231 > 231 __asm("movq %%gs:%1,%0" : "=r" (td) > (kgdb) frame 6 > #6 0xffffffff80736cec in in_pcblookup_local (pcbinfo=0xffffffff80dc9180, laddr= > {s_addr = 50374848}, lport=339, lookupflags=1, cred=0xfffffe016cdad100) > at /usr/src/sys/netinet/in_pcb.c:1438 > 1438 LIST_FOREACH(phd, porthash, phd_hash) { > (kgdb) list *0xffffffff80736cec > 0xffffffff80736cec is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:1439). > 1434 * port hash list. > 1435 */ > 1436 porthash = &pcbinfo->ipi_porthashbase[INP_PCBPORTHASH(lport, > 1437 pcbinfo->ipi_porthashmask)]; > 1438 LIST_FOREACH(phd, porthash, phd_hash) { > 1439 if (phd->phd_port == lport) > 1440 break; > 1441 } > 1442 if (phd != NULL) { > 1443 /* Can you see what 'phd' and 'porthash' are? If kgdb can't see them you can reconstruct what 'porthash' should be (you know the lport arg to this routine and can get the relevant fields from 'pcbinfo'). -- John Baldwin