From owner-freebsd-atm@FreeBSD.ORG Mon Nov 10 15:17:26 2008 Return-Path: Delivered-To: freebsd-atm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 414461065670; Mon, 10 Nov 2008 15:17:26 +0000 (UTC) (envelope-from scaron@umich.edu) Received: from sonnet.diablonet.net (sonnet.diablonet.net [75.144.70.42]) by mx1.freebsd.org (Postfix) with ESMTP id E1E608FC24; Mon, 10 Nov 2008 15:17:25 +0000 (UTC) (envelope-from scaron@umich.edu) Received: from [141.211.10.207] (host10-207.sph.umich.edu [141.211.10.207]) by sonnet.diablonet.net (Postfix) with ESMTP id 8CE343DD066; Mon, 10 Nov 2008 09:46:32 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v753.1) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <4098EE5C-1B7E-42B5-8F6C-F35DB8D4C917@umich.edu> Content-Transfer-Encoding: 7bit From: Sean Caron Date: Mon, 10 Nov 2008 09:46:46 -0500 To: freebsd-atm@freebsd.org, freebsd-net@freebsd.org X-Mailer: Apple Mail (2.753.1) Cc: Subject: Occasional kernel panic + reboot on 7.0-RELEASE, sparc64, fatm card. X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Nov 2008 15:17:26 -0000 Hi folks, I posted this originally to the Freebsd/sparc64 general mailing list and someone there suggested that I send it this way, with the following note. "This apparently is a NULL-pointer dereference (probably "m" in sbsndptr()), with the cause being in one of the stacks involved. I'd suggest to report this backtrace to the atm@ and net@ lists." Quick background - I'm using fatm on FreeBSD/sparc64 7.0-RELEASE with a FORE PCA-200E PCI ATM card (fatm). I am using the Cranor (natm) driver. It generally works well but every couple of weeks the system will kernel panic and reboot. I switched on kernel dumps on panic and here's what I got (this time): sonnet.diablonet.net> kgdb kernel.debug /var/crash/vmcore.0 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions.. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "sparc64-marcel-freebsd". Unread portion of the kernel message buffer: panic: trap: fast data access mmu miss Uptime: 16d13h9m7s Dumping 1024 MB (2 chunks) chunk at 0: 536870912 bytes | #0 0x00000000c0280cd8 in doadump () at /usr/src/sys/kern/ kern_shutdown.c:240 240 savectx(&dumppcb); (kgdb) backtrace #0 0x00000000c0280cd8 in doadump () at /usr/src/sys/kern/ kern_shutdown.c:240 #1 0x00000000c0281608 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0x00000000c0281860 in panic (fmt=0xc066c6e0 "trap: %s") at /usr/src/sys/kern/kern_shutdown.c:563 #3 0x00000000c0541de4 in trap (tf=0xe5390e50) at /usr/src/sys/sparc64/sparc64/trap.c:378 #4 0x00000000c0070fe0 in tl1_trap () #5 0x00000000c02dd1d0 in sbsndptr (sb=0xfffff800014be6f0, off=0, len=1390, moff=0xe5391064) at /usr/src/sys/kern/uipc_sockbuf.c:939 #6 0x00000000c03edac4 in tcp_output (tp=0xfffff800014be6f0) at /usr/src/sys/netinet/tcp_output.c:802 #7 0x00000000c03edac4 in tcp_output (tp=0xfffff800014fce38) at /usr/src/sys/netinet/tcp_output.c:802 #8 0x00000000c03eaf98 in tcp_do_segment (m=0xfffff8005b354000, th=0xfffff8000133283c, so=0xfffff800014be570, tp=0xfffff800014fce38, drop_hdrlen=52, tlen=0) at /usr/src/sys/netinet/tcp_input.c:2347 #9 0x00000000c03ec214 in tcp_input (m=0xfffff8005b354000, off0=Variable "off0" is not available. ) at /usr/src/sys/netinet/tcp_input.c:845 #10 0x00000000c0381128 in ip_input (m=0xfffff8005b354000) at /usr/src/sys/netinet/ip_input.c:665 #11 0x00000000c0339cd0 in netisr_dispatch (num=2, m=0xfffff8005b354000) at /usr/src/sys/net/netisr.c:185 #12 0x00000000c032a930 in atm_input (ifp=0xfffff8000103c000, ah=0xe539162c, m=0xfffff8005b354000, rxhand=0x0) at /usr/src/sys/net/ if_atmsubr.c:347 #13 0x00000000c013d410 in fatm_intr (p=0xfffff80001173c00) at /usr/src/sys/dev/fatm/if_fatm.c:1573 #14 0x00000000c02615ec in ithread_loop (arg=0xfffff800011ce760) at /usr/src/sys/kern/kern_intr.c:1036 #15 0x00000000c025dd54 in fork_exit (callout=0xc0261420 , arg=0xfffff800011ce760, frame=0xe5391880) at /usr/src/sys/kern/kern_fork.c:781 #16 0x00000000c00711d0 in fork_trampoline () #17 0x00000000c00711d0 in fork_trampoline () Previous frame identical to this frame (corrupt stack?) (kgdb) up 15 #15 0x00000000c025dd54 in fork_exit (callout=0xc0261420 , arg=0xfffff800011ce760, frame=0xe5391880) at /usr/src/sys/kern/kern_fork.c:781 781 callout(arg, frame); (kgdb) list 776 * cpu_set_fork_handler intercepts this function call to 777 * have this call a non-return function to stay in kernel mode. 778 * initproc has its own fork handler, but it does return. 779 */ 780 KASSERT(callout != NULL, ("NULL callout in fork_exit")); 781 callout(arg, frame); 782 783 /* 784 * Check if a kernel thread misbehaved and returned from its main 785 * function. (kgdb) down #14 0x00000000c02615ec in ithread_loop (arg=0xfffff800011ce760) at /usr/src/sys/kern/kern_intr.c:1036 1036 ih->ih_handler(ih->ih_argument); (kgdb) list 1031 __func__, p->p_pid, (void *)ih->ih_handler, 1032 ih->ih_argument, ih->ih_name, ih->ih_flags); 1033 1034 if (!(ih->ih_flags & IH_MPSAFE)) 1035 mtx_lock(&Giant); 1036 ih->ih_handler(ih->ih_argument); 1037 if (!(ih->ih_flags & IH_MPSAFE)) 1038 mtx_unlock(&Giant); 1039 } 1040 if (!(ie->ie_flags & IE_SOFT)) (kgdb) down #13 0x00000000c013d410 in fatm_intr (p=0xfffff80001173c00) at /usr/src/sys/dev/fatm/if_fatm.c:1573 1573 atm_input(ifp, &aph, m0, vc->rxhand); (kgdb) list 1568 ifp->if_ipackets++; 1569 1570 vc->ipackets++; 1571 vc->ibytes += m0->m_pkthdr.len; 1572 1573 atm_input(ifp, &aph, m0, vc->rxhand); 1574 } 1575 1576 H_SETSTAT(q->q.statp, FATM_STAT_FREE); 1577 H_SYNCSTAT_PREWRITE(sc, q->q.statp); (kgdb) down #12 0x00000000c032a930 in atm_input (ifp=0xfffff8000103c000, ah=0xe539162c, m=0xfffff8005b354000, rxhand=0x0) at /usr/src/sys/net/ if_atmsubr.c:347 347 netisr_dispatch(isr, m); (kgdb) list 342 else 343 m_freem(m); 344 return; 345 } 346 } 347 netisr_dispatch(isr, m); 348 } 349 350 /* 351 * Perform common duties while attaching to interface list. (kgdb) down #11 0x00000000c0339cd0 in netisr_dispatch (num=2, m=0xfffff8005b354000) at /usr/src/sys/net/netisr.c:185 185 ni->ni_handler(m); (kgdb) list 180 * the packet but now do not. Doing so here will 181 * not preserve ordering so instead we fallback to 182 * guaranteeing order only from dispatch points 183 * in the system (see above). 184 */ 185 ni->ni_handler(m); 186 } else { 187 isrstat.isrs_deferred++; 188 if (IF_HANDOFF(ni->ni_queue, m, NULL)) 189 schednetisr(num); (kgdb) down #10 0x00000000c0381128 in ip_input (m=0xfffff8005b354000) at /usr/src/sys/netinet/ip_input.c:665 665 (*inetsw[ip_protox[ip->ip_p]].pr_input)(m, hlen); (kgdb) list 660 /* 661 * Switch out to protocol's input routine. 662 */ 663 ipstat.ips_delivered++; 664 665 (*inetsw[ip_protox[ip->ip_p]].pr_input)(m, hlen); 666 return; 667 bad: 668 m_freem(m); 669 } (kgdb) down #9 0x00000000c03ec214 in tcp_input (m=0xfffff8005b354000, off0=Variable "off0" is not available. ) at /usr/src/sys/netinet/tcp_input.c:845 845 tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen); (kgdb) list 840 /* 841 * Segment belongs to a connection in SYN_SENT, ESTABLISHED or later 842 * state. tcp_do_segment() always consumes the mbuf chain, unlocks 843 * the inpcb, and unlocks pcbinfo. 844 */ 845 tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen); 846 INP_INFO_UNLOCK_ASSERT(&tcbinfo); 847 return; 848 849 dropwithreset: (kgdb) down #8 0x00000000c03eaf98 in tcp_do_segment (m=0xfffff8005b354000, th=0xfffff8000133283c, so=0xfffff800014be570, tp=0xfffff800014fce38, drop_hdrlen=52, tlen=0) at /usr/src/sys/netinet/tcp_input.c:2347 2347 (void) tcp_output(tp); (kgdb) list 2342 2343 /* 2344 * Return any desired output. 2345 */ 2346 if (needoutput || (tp->t_flags & TF_ACKNOW)) 2347 (void) tcp_output(tp); 2348 2349 check_delack: 2350 KASSERT(headlocked == 0, ("%s: check_delack: head locked", 2351 __func__)); (kgdb) down #7 0x00000000c03edac4 in tcp_output (tp=0xfffff800014fce38) at /usr/src/sys/netinet/tcp_output.c:802 802 mb = sbsndptr(&so->so_snd, off, len, &moff); (kgdb) list 797 798 /* 799 * Start the m_copy functions from the closest mbuf 800 * to the offset in the socket buffer chain. 801 */ 802 mb = sbsndptr(&so->so_snd, off, len, &moff); 803 804 if (len <= MHLEN - hdrlen - max_linkhdr) { 805 m_copydata(mb, moff, (int)len, 806 mtod(m, caddr_t) + hdrlen); (kgdb) down #6 0x00000000c03edac4 in tcp_output (tp=0xfffff800014be6f0) at /usr/src/sys/netinet/tcp_output.c:802 802 mb = sbsndptr(&so->so_snd, off, len, &moff); (kgdb) list 797 798 /* 799 * Start the m_copy functions from the closest mbuf 800 * to the offset in the socket buffer chain. 801 */ 802 mb = sbsndptr(&so->so_snd, off, len, &moff); 803 804 if (len <= MHLEN - hdrlen - max_linkhdr) { 805 m_copydata(mb, moff, (int)len, 806 mtod(m, caddr_t) + hdrlen); (kgdb) down #5 0x00000000c02dd1d0 in sbsndptr (sb=0xfffff800014be6f0, off=0, len=1390, moff=0xe5391064) at /usr/src/sys/kern/uipc_sockbuf.c:939 939 off > 0 && off >= m->m_len; (kgdb) list 934 *moff = off - sb->sb_sndptroff; 935 m = ret = sb->sb_sndptr ? sb->sb_sndptr : sb->sb_mb; 936 937 /* Advance by len to be as close as possible for the next transmit. */ 938 for (off = off - sb->sb_sndptroff + len - 1; 939 off > 0 && off >= m->m_len; 940 m = m->m_next) { 941 sb->sb_sndptroff += m->m_len; 942 off -= m->m_len; 943 } (kgdb) down #4 0x00000000c0070fe0 in tl1_trap () (kgdb) list 944 sb->sb_sndptr = m; 945 946 return (ret); 947 } 948 949 /* 950 * Drop a record off the front of a sockbuf and move the next record to the 951 * front. 952 */ 953 void (kgdb) quit sonnet.diablonet.net> Please let me know if further information is required and I will furnish, no problem. Thanks, -Sean