Date: Thu, 14 Dec 2006 08:41:08 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Alex Povolotsky <tarkhil@webmail.sub.ru> Cc: freebsd-stable@freebsd.org Subject: Re: Strange panics Message-ID: <20061214083924.C72681@fledge.watson.org> In-Reply-To: <457FA5D7.9060502@webmail.sub.ru> References: <457FA5D7.9060502@webmail.sub.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 13 Dec 2006, Alex Povolotsky wrote:
> I'm getting lots of panics on quite lightly loaded box under 6.1-RELEASE.
>
> The box runned two jails without any flaws for several weeks, after adding
> third one it begins to panics at least once a day.
>
> I have no idea on what could cause this.
>
> Here is my dmesg.boot (partially, I can send it all)
This problem should be fixed in 6.2-RC1 and forward, I believe. If you're
unable to upgrade, I may be able to provide patch backports, but would
recommend an upgrade generally. This panic occurs because of a race condition
in reset handling with respect to socket options, and tends to occur more
frequently on boxes with long-lived connections that get reset and
applications that frequently check or set socket options.
Robert N M Watson
Computer Laboratory
University of Cambridge
>
>
>
> Copyright (c) 1992-2006 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD 6.1-RELEASE #0: Tue Dec 12 08:44:28 MSK 2006
> tarkhil@box2s.sub.ru:/usr/obj/usr/src/sys/SMP
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.01-MHz 686-class CPU)
> Origin = "GenuineIntel" Id = 0xf43 Stepping = 3
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE>
> Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,<b14>>
> AMD Features=0x20100000<NX,LM>
> Logical CPUs per core: 2
> real memory = 2147352576 (2047 MB)
> avail memory = 2096328704 (1999 MB)
> ACPI APIC Table: <A M I OEMAPIC >
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
> cpu2 (AP): APIC ID: 6
> cpu3 (AP): APIC ID: 7
>
> and here is kgbd output
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 06
> fault virtual address = 0xac
> fault code = supervisor write, page not present
> instruction pointer = 0x20:0xc06ee9d0
> stack pointer = 0x28:0xe91e2b18
> frame pointer = 0x28:0xe91e2b34
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 16260 (tcpserver)
> trap number = 12
> panic: page fault
> cpuid = 2
> Uptime: 19h33m1s
>
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc06507c9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
> #2 0xc0650af1 in panic (fmt=0xc08958ae "%s") at
> /usr/src/sys/kern/kern_shutdown.c:558
> #3 0xc084a2cc in trap_fatal (frame=0xe91e2ad8, eva=172) at
> /usr/src/sys/i386/i386/trap.c:836
> #4 0xc084a00b in trap_pfault (frame=0xe91e2ad8, usermode=0, eva=172) at
> /usr/src/sys/i386/i386/trap.c:744
> #5 0xc0849c45 in trap (frame=
> {tf_fs = -1065156600, tf_es = -1056636888, tf_ds = 40, tf_edi = 55,
> tf_esi = 0, tf_ebp = -383898828, tf_isp = -383898876, tf_ebx = -383898480,
> tf_edx = -944133120, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2,
> tf_eip = -1066473008, tf_cs = 32, tf_eflags = 66182, tf_esp = -880992256,
> tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:434
> #6 0xc0836c4a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
> #7 0xc06ee9d0 in ip_ctloutput (so=0x4, sopt=0xe91e2c90) at atomic.h:146
> #8 0xc06feb73 in tcp_ctloutput (so=0xc811b6f4, sopt=0xe91e2c90) at
> /usr/src/sys/netinet/tcp_usrreq.c:1038
> #9 0xc068bdf8 in sosetopt (so=0xc811b6f4, sopt=0xe91e2c90) at
> /usr/src/sys/kern/uipc_socket.c:1560
> #10 0xc0691115 in kern_setsockopt (td=0xc7b9ac00, s=0, level=4, name=4,
> val=0xc7b9ac00, valseg=UIO_USERSPACE, valsize=0)
> at /usr/src/sys/kern/uipc_syscalls.c:1351
> #11 0xc0691046 in setsockopt (td=0xc7b9ac00, uap=0x4) at
> /usr/src/sys/kern/uipc_syscalls.c:1307
> #12 0xc084a613 in syscall (frame=
> {tf_fs = -1078001605, tf_es = 59, tf_ds = -1078001605, tf_edi =
> -1077942204, tf_esi = 3, tf_ebp = -1077942408, tf_isp = -383898268, tf_ebx =
> 0, tf_edx = 2, tf_ecx = 134545464, tf_eax = 105, tf_trapno = 12, tf_err = 2,
> tf_eip = 186394799, tf_cs = 51, tf_eflags = 642, tf_esp = -1077942452, tf_ss
> = 59}) at /usr/src/sys/i386/i386/trap.c:981
> #13 0xc0836c9f in Xint0x80_syscall () at
> /usr/src/sys/i386/i386/exception.s:200
> #14 0x00000033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
>
>
> === cut ===
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0xac
> fault code = supervisor write, page not present
> instruction pointer = 0x20:0xc06ee9d0
> stack pointer = 0x28:0xe8f77b18
> frame pointer = 0x28:0xe8f77b34
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 86881 (tcpserver)
> trap number = 12
> panic: page fault
> cpuid = 0
> Uptime: 1h55m49s
> Dumping 2047 MB (2 chunks)
> chunk 0: 1MB (158 pages) ... ok
> chunk 1: 2047MB (524000 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903
> 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663
> 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423
> 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183
> 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927
> 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623
> 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319
> 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
>
> #0 doadump () at pcpu.h:165
> 165 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc06507c9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
> #2 0xc0650af1 in panic (fmt=0xc08958ae "%s") at
> /usr/src/sys/kern/kern_shutdown.c:558
> #3 0xc084a2cc in trap_fatal (frame=0xe8f77ad8, eva=172) at
> /usr/src/sys/i386/i386/trap.c:836
> #4 0xc084a00b in trap_pfault (frame=0xe8f77ad8, usermode=0, eva=172) at
> /usr/src/sys/i386/i386/trap.c:744
> #5 0xc0849c45 in trap (frame=
> {tf_fs = -939130872, tf_es = -1056636888, tf_ds = 40, tf_edi = 55,
> tf_esi = 0, tf_ebp = -386434252, tf_isp = -386434300, tf_ebx = -386433904,
> tf_edx = -951618816, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2,
> tf_eip = -1066473008, tf_cs = 32, tf_eflags = 66182, tf_esp = -953071872,
> tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:434
> #6 0xc0836c4a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
> #7 0xc06ee9d0 in ip_ctloutput (so=0x4, sopt=0xe8f77c90) at atomic.h:146
> #8 0xc06feb73 in tcp_ctloutput (so=0xc8c0142c, sopt=0xe8f77c90) at
> /usr/src/sys/netinet/tcp_usrreq.c:1038
> #9 0xc068bdf8 in sosetopt (so=0xc8c0142c, sopt=0xe8f77c90) at
> /usr/src/sys/kern/uipc_socket.c:1560
> #10 0xc0691115 in kern_setsockopt (td=0xc7477300, s=0, level=4, name=4,
> val=0xc7477300, valseg=UIO_USERSPACE, valsize=0)
> at /usr/src/sys/kern/uipc_syscalls.c:1351
> #11 0xc0691046 in setsockopt (td=0xc7477300, uap=0x4) at
> /usr/src/sys/kern/uipc_syscalls.c:1307
> #12 0xc084a613 in syscall (frame=
> {tf_fs = -1078001605, tf_es = 59, tf_ds = -1078001605, tf_edi =
> -1077942088, tf_esi = 3, tf_ebp = -1077942296, tf_isp = -386433692, tf_ebx =
> 0, tf_edx = 2, tf_ecx = 134545464, tf_eax = 105, tf_trapno = 12, tf_err = 2,
> tf_eip = 186394799, tf_cs = 51, tf_eflags = 646, tf_esp = -1077942340, tf_ss
> = 59}) at /usr/src/sys/i386/i386/trap.c:981
> #13 0xc0836c9f in Xint0x80_syscall () at
> /usr/src/sys/i386/i386/exception.s:200
> #14 0x00000033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
>
> === cut ===
>
> NIC is Intel (em).
>
> Can anyone provide any help with this?
>
> Alex.
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061214083924.C72681>