Date: Mon, 18 Feb 2008 13:45:20 -0500 From: "Rob Watt" <rob@hudson-trading.com> To: freebsd-net@freebsd.org Subject: panic in 6.3-RELEASE when multi-cast client exits Message-ID: <cf6c7840802181045k7ccad96m50c547aed687e14@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi. We recently upgraded some of our machines to 6.3-RELEASE and we have been plagued by repeatable panics when our multi-cast client applications exit. Our machines have Intel X5365 processors, LSI MegaSAS 1064R cards, and Inte= l Pro 1000 MF nic cards (although we have seen this problem with the onboard Intel copper nics as well). We have seen this panic with machines that have Tyan boards as well as Super Micro. I have seen a few postings that seem to refer to related panics, and bug http://www.freebsd.org/cgi/query-pr.cgi?pr=3D116077 contains a patch that seems like it should address the problem, but our patched system still panics. I have attached the output from 3 of the dumps/backtraces. Dump #1 is probably the most useful. I am happy to provide more info if necessary. dump/bt #1: Fatal trap 9: general protection fault while in kernel mode cpuid =3D 0; apic id =3D 00 instruction pointer =3D 0x8:0xffffffff8033e899 stack pointer =3D 0x10:0xffffffffb98b9890 frame pointer =3D 0x10:0xffffff001e919840LSI MegaSAS 1064R code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 71509 (ctsfeed) trap number =3D 9 panic: general protection fault cpuid =3D 3 Uptime: 1d22h17m7s Dumping 8190 MB (3 chunks) ... ... #0 doadump () at pcpu.h:172 172 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:172 #1 0x0000000000000004 in ?? () #2 0xffffffff8029ab57 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff8029b1f1 in panic (fmt=3D0xffffff01b5767980 "X3\221=B5\001=FF=FF=FF`\003pB") at /usr/src/sys/kern/kern_shutdown.c:565 #4 0xffffffff8042bc8f in trap_fatal (frame=3D0xffffff01b5767980, eva=3D18446742981539083096) at /usr/src/sys/amd64/amd64/trap.c:669 #5 0xffffffff8042c192 in trap (frame=3D {tf_rdi =3D -1098998769600, tf_rsi =3D -1092172220032, tf_rdx =3D 0, = tf_rcx =3D 1, tf_r8 =3D 4, tf_r9 =3D -1097974286056, tf_rax =3D 0, tf_rbx =3D -109= 2074318848, tf_rbp =3D -1098998769600, tf_r10 =3D -1098398611680, tf_r11 =3D 4, tf_r12 = =3D 7956010471516012544, tf_r13 =3D -2140970848, tf_r14 =3D 0, tf_r15 =3D -1092172220032, tf_trapno =3D 9, tf_addr =3D 0, tf_flags =3D -2140970776, t= f_err =3D 0, tf_rip =3D -2144081767, tf_cs =3D 8, tf_rflags =3D 66182, tf_rsp =3D -11= 82033760, tf_ss =3D 16}) at /usr/src/sys/amd64/amd64/trap.c:470 #6 0xffffffff8041333b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:168 #7 0xffffffff8033e899 in in_delmulti (inm=3D0xffffff001e919840) at /usr/src/sys/netinet/in.c:1063 #8 0xffffffff80348636 in ip_freemoptions (imo=3D0xffffff01bb4c5400) at /usr/src/sys/netinet/ip_output.c:2064 #9 0xffffffff803412e0 in in_pcbdetach (inp=3D0xffffff00a5163000) at /usr/src/sys/netinet/in_pcb.c:718 #10 0xffffffff8035c8b3 in udp_detach (so=3D0xffffff00a5163000) at /usr/src/sys/netinet/udp_usrreq.c:1071 #11 0xffffffff802e4f10 in soclose (so=3D0xffffff018ca7ec08) at /usr/src/sys/kern/uipc_socket.c:459 #12 0xffffffff802cfa75 in soo_close (fp=3D0xffffff0010074ca8, td=3D0xffffff01b5767980) at /usr/src/sys/kern/sys_socket.c:317 #13 0xffffffff8026dc7e in fdrop_locked (fp=3D0xffffff0010074ca8, td=3D0xffffff01b5767980) at file.h:296 #14 0xffffffff8026e1b5 in closef (fp=3D0xffffff0010074ca8, td=3D0xffffff01b5767980) at /usr/src/sys/kern/kern_descrip.c:1933 #15 0xffffffff80270228 in fdfree (td=3D0xffffff01b5767980) at /usr/src/sys/kern/kern_descrip.c:1651 #16 0xffffffff8027a9d1 in exit1 (td=3D0xffffff01b5767980, rv=3D256) at /usr/src/sys/kern/kern_exit.c:273 #17 0xffffffff8027bb8e in sys_exit (td=3D0xffffff001e919840, uap=3D0xffffff01b5767980) at /usr/src/sys/kern/kern_exit.c:99 #18 0xffffffff8042cb81 in syscall (frame=3D {tf_rdi =3D 1, tf_rsi =3D 34365343744, tf_rdx =3D 34371641456, tf_rcx= =3D 10, tf_r8 =3D 0, tf_r9 =3D 0, tf_rax =3D 1, tf_rbx =3D 1, tf_rbp =3D 1407374883= 45952, tf_r10 =3D 2, tf_r11 =3D 534, tf_r12 =3D 167, tf_r13 =3D 0, tf_r14 =3D 0, t= f_r15 =3D 140737488349744, tf_trapno =3D 12, tf_addr =3D 34372863240, tf_flags =3D 140737488344048, tf_err =3D 2, tf_rip =3D 34371644108, tf_cs =3D 43, tf_rfl= ags =3D 518, tf_rsp =3D 140737488343416, tf_ss =3D 35}) at /usr/src/sys/amd64/amd64/trap.c:807 #19 0xffffffff80413538 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:287 #20 0x0000000800b5aacc in ?? () dump/bt #2: Fatal trap 9: general protection fault while in kernel mode cpuid =3D 5; apic id =3D 05 instruction pointer =3D 0x8:0xffffffff8034627e stack pointer =3D 0x10:0xffffffffb6ef1ad0 frame pointer =3D 0x10:0x14 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 20 (swi1: net) trap number =3D 9 panic: general protection fault cpuid =3D 5 Uptime: 23h43m21s Dumping 8190 MB (3 chunks) ... ... #0 doadump () at pcpu.h:172 172 pcpu.h: No such file or directory. in pcpu.h #0 doadump () at pcpu.h:172 #1 0x0000000000000004 in ?? () #2 0xffffffff8029ab57 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff8029b1f1 in panic (fmt=3D0xffffff021ef0c720 "X=D3=F0\036\002=FF=FF=FF\020=C5=F4\036\002=FF=FF=FF") at /usr/src/sys/kern= /kern_shutdown.c:565 #4 0xffffffff8042bc8f in trap_fatal (frame=3D0xffffff021ef0c720, eva=3D18446742983306957656) at /usr/src/sys/amd64/amd64/trap.c:669 #5 0xffffffff8042c192 in trap (frame=3D {tf_rdi =3D -1096796658432, tf_rsi =3D -1090402597088, tf_rdx =3D -1099077979968, tf_rcx =3D 8390317583334711354, tf_r8 =3D -1098648831624, t= f_r9 =3D 3916890112, tf_rax =3D 1, tf_rbx =3D -1097507680240, tf_rbp =3D 20, tf_= r10 =3D -1090404108288, tf_r11 =3D 100, tf_r12 =3D -1098648832000, tf_r13 =3D 4, tf= _r14 =3D -1099498930176, tf_r15 =3D -1096796658432, tf_trapno =3D 9, tf_addr =3D 0, tf_flags =3D -1096796658432, tf_err =3D 0, tf_rip =3D -2144050562, tf_cs = =3D 8, tf_rflags =3D 66182, tf_rsp =3D -1225844000, tf_ss =3D 16}) at /usr/src/sys/amd64/amd64/trap.c:470 #6 0xffffffff8041333b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:168 #7 0xffffffff8034627e in ip_input (m=3D0xffffff00a1d32500) at /usr/src/sys/netinet/ip_input.c:624 #8 0xffffffff803302cc in netisr_processqueue (ni=3D0xffffffff80633f90) at /usr/src/sys/net/netisr.c:236 #9 0xffffffff8033057d in swi_net (dummy=3D0xffffff00a1d32500) at /usr/src/sys/net/netisr.c:349 #10 0xffffffff8027fe38 in ithread_loop (arg=3D0xffffff0000090520) at /usr/src/sys/kern/kern_intr.c:682 #11 0xffffffff8027e5d7 in fork_exit (callout=3D0xffffffff8027fcf0 <ithread_loop>, arg=3D0xffffff0000090520, frame=3D0xffffffffb6ef1c50) at /usr/src/sys/kern/kern_fork.c:788 #12 0xffffffff804136fe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:411 #13 0x0000000000000000 in ?? () #14-125 garbage dump/bt #3: Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x1 fault code =3D supervisor read data, page not present instruction pointer =3D 0x8:0xffffffff8034627e stack pointer =3D 0x10:0xffffffffb6ef1ad0 frame pointer =3D 0x10:0x14 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 20 (swi1: net) trap number =3D 12 panic: page fault cpuid =3D 0 Uptime: 1d22h10m53s Dumping 8190 MB (3 chunks) ... ... #0 sched_switch (td=3D0xffffff01b85114c0, newtd=3D0xffffff021ef49720, flag= s=3D1) at /usr/src/sys/kern/sched_4bsd.c:980 980 sched_lock.mtx_lock =3D (uintptr_t)td; (kgdb) bt #0 sched_switch (td=3D0xffffff01b85114c0, newtd=3D0xffffff021ef49720, flag= s=3D1) at /usr/src/sys/kern/sched_4bsd.c:980 #1 0x0000000000000246 in ?? () #2 0xffffffff8028fe25 in _mtx_lock_spin (m=3D0xffffff022c50d728, tid=3D0, opts=3D0, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_mutex.c:638 #3 0x0000000000000004 in ?? () #4 0x0000000000000000 in ?? () #5 0xffffffff803f1bbb in uma_zfree_arg (zone=3D0xffffffff802c5ac6, item=3D0xffffff01b85114c0, udata=3D0xffffff01b8a746b0) at /usr/src/sys/vm/uma_core.c:2297 #6 0xffffff01b4f6ca80 in ?? () #7 0xffffff01b755a460 in ?? () #8 0xffffff003bb3a000 in ?? () #9 0x0000000000000020 in ?? () #10 0xffffff01b4f6ca80 in ?? () #11 0xffffff01b755a460 in ?? () #12 0xffffff0000bce000 in ?? () #13 0x0000000000000013 in ?? () #14 0xffffff01b77f2000 in ?? () #15 0x0000000000000000 in ?? () #16 0xffffffff802a5b88 in binuptime (bt=3D0x0) at /usr/src/sys/kern/kern_tc.c:143 #17 0xffffffff802a8a23 in thread_exit () at /usr/src/sys/kern/kern_thread.c:597 #18 0xffffffff8027b227 in exit1 (td=3D0xffffff01b4f6ca80, rv=3D-2143347781)= at /usr/src/sys/kern/kern_exit.c:528 #19 0xffffffff8027bb8e in sys_exit (td=3D0x0, uap=3D0x0) at /usr/src/sys/kern/kern_exit.c:99 #20 0xffffffff8042cb81 in syscall (frame=3D {tf_rdi =3D 0, tf_rsi =3D 0, tf_rdx =3D 4285347, tf_rcx =3D 6, tf_r8 = =3D 0, tf_r9 =3D 0, tf_rax =3D 1, tf_rbx =3D 3, tf_rbp =3D 140737488350784, tf_r10= =3D 0, tf_r11 =3D 0, tf_r12 =3D 140737488350752, tf_r13 =3D 0, tf_r14 =3D 0, tf_r1= 5 =3D 0, tf_trapno =3D 22, tf_addr =3D 0, tf_flags =3D 0, tf_err =3D 2, tf_rip =3D 3= 4369792716, tf_cs =3D 43, tf_rflags =3D 518, tf_rsp =3D 140737488350312, tf_ss =3D 35})= at /usr/src/sys/amd64/amd64/trap.c:807 #21 0xffffffff80413538 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:287 #22 0x0000000800996acc in ?? () Thanks --=20 Rob Watt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cf6c7840802181045k7ccad96m50c547aed687e14>