Date: Wed, 22 Jul 2015 13:53:53 +0300 From: "Zeus Panchenko" <zeus@ibs.dn.ua> To: <freebsd-stable@FreeBSD.org> Subject: [panic] 10.1-RELEASE-p14 #4 r285499M sporadic reboot Message-ID: <20150722135353.70269@smtp.new-ukraine.org>
next in thread | raw e-mail | index | archive | help
=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
hi,
on important production server I've started to experience weird regular
(once a day-two) sporadic reboots after last upgrade (2015-07-17), and
would appreciate if someone could help me to analyze further and fix the
issue.
Collected crash information below
full logs are available upon request.
Thank you!
backtrace:
=2D ---[ quotation start ]-------------------------------------------
fo02.ibs dumped core - see /var/crash/vmcore.2
Wed Jul 22 08:01:15 EEST 2015
FreeBSD fo02.ibs 10.1-RELEASE-p14 FreeBSD 10.1-RELEASE-p14 #4 r285499M: Fri=
Jul 17 22:45:54 EEST 2015 root@fo02.ibs:/usr/obj/usr/src/sys/FO02.2015=
0307233500 amd64
panic: page fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid =3D 3; apic id =3D 06
fault virtual address =3D 0xffff80403c82b740
fault code =3D supervisor read data, page not present
instruction pointer =3D 0x20:0xffffffff80730e9c
stack pointer =3D 0x28:0xfffffe000023d790
frame pointer =3D 0x28:0xfffffe000023d7a0
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 12 (irq283: igb3:que 3)
trap number =3D 12
panic: page fault
cpuid =3D 3
KDB: stack backtrace:
#0 0xffffffff804fe970 at kdb_backtrace+0x60
#1 0xffffffff804c52a5 at panic+0x155
#2 0xffffffff8073f1cf at trap_fatal+0x38f
#3 0xffffffff8073f4e8 at trap_pfault+0x308
#4 0xffffffff8073eb4a at trap+0x47a
#5 0xffffffff80724cb2 at calltrap+0x8
#6 0xffffffff807910cb at bounce_bus_dmamap_load_buffer+0x1bb
#7 0xffffffff804f9c82 at bus_dmamap_load_mbuf_sg+0x72
#8 0xffffffff810c37fe at igb_refresh_mbufs+0x19e
#9 0xffffffff810c3603 at igb_rxeof+0x683
#10 0xffffffff810c3ab4 at igb_msix_que+0xe4
#11 0xffffffff80498e1b at intr_event_execute_handlers+0xab
#12 0xffffffff80499266 at ithread_loop+0x96
#13 0xffffffff80496a3a at fork_exit+0x9a
#14 0xffffffff807251ee at fork_trampoline+0xe
Uptime: 1d10h23m19s
Dumping 1191 out of 8136 MB:..2%..11%..21%..31%..41%..52%..61%..72%..81%..9=
2%
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/pf.ko.symbols...done.
Loaded symbols for /boot/kernel/pf.ko.symbols
Reading symbols from /boot/kernel/if_igb.ko.symbols...done.
Loaded symbols for /boot/kernel/if_igb.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/aesni.ko.symbols...done.
Loaded symbols for /boot/kernel/aesni.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/pflog.ko.symbols...done.
Loaded symbols for /boot/kernel/pflog.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
Loaded symbols for /boot/kernel/if_bridge.ko.symbols
Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
Loaded symbols for /boot/kernel/bridgestp.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
#0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
#1 0xffffffff804c4f22 in kern_reboot (howto=3D260)
at /usr/src/sys/kern/kern_shutdown.c:452
#2 0xffffffff804c52e4 in panic (fmt=3D<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff8073f1cf in trap_fatal (frame=3D<value optimized out>,=20
eva=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4 0xffffffff8073f4e8 in trap_pfault (frame=3D0xfffffe000023d6e0,=20
usermode=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5 0xffffffff8073eb4a in trap (frame=3D0xfffffe000023d6e0)
at /usr/src/sys/amd64/amd64/trap.c:440
#6 0xffffffff80724cb2 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:232
#7 0xffffffff80730e9c in pmap_kextract (va=3D729286318152418877)
at /usr/src/sys/amd64/amd64/pmap.c:639
#8 0xffffffff807910cb in bounce_bus_dmamap_load_buffer (
dmat=3D0xfffff800034c1800, map=3D0xffffffff80be01e8,=20
buf=3D<value optimized out>, buflen=3D<value optimized out>,=20
pmap=3D0xffffffff80c52ce0, flags=3D<value optimized out>,=20
segs=3D<value optimized out>) at /usr/src/sys/x86/x86/busdma_bounce.c:6=
90
#9 0xffffffff804f9c82 in bus_dmamap_load_mbuf_sg (dmat=3D0xfffff800034c180=
0,=20
map=3D0x0, m0=3D<value optimized out>, segs=3D0xfffffe000023d8b0,=20
nsegs=3D0xfffffe000023d8ac, flags=3D<value optimized out>)
at /usr/src/sys/kern/subr_bus_dma.c:110
#10 0xffffffff810c37fe in igb_refresh_mbufs (rxr=3D0xfffff8000349f330,=20
limit=3D2101) at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4144
#11 0xffffffff810c3603 in igb_rxeof (count=3D<value optimized out>)
at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4998
#12 0xffffffff810c3ab4 in igb_msix_que (arg=3D0xfffff800034a8b38)
at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:1621
#13 0xffffffff80498e1b in intr_event_execute_handlers (
p=3D<value optimized out>, ie=3D0xfffff80003473c00)
at /usr/src/sys/kern/kern_intr.c:1263
#14 0xffffffff80499266 in ithread_loop (arg=3D0xfffff800034a53e0)
at /usr/src/sys/kern/kern_intr.c:1276
#15 0xffffffff80496a3a in fork_exit (
callout=3D0xffffffff804991d0 <ithread_loop>, arg=3D0xfffff800034a53e0,=
=20
frame=3D0xfffffe000023dac0) at /usr/src/sys/kern/kern_fork.c:996
#16 0xffffffff807251ee in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:606
#17 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)=20
=2D ---[ quotation end ]-------------------------------------------
#> uname -a
FreeBSD 10.1-RELEASE-p14 #4 r285499M amd64
#> dmesg
...
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (3100.06-MHz K8-class CPU)
...
real memory =3D 8589934592 (8192 MB)
avail memory =3D 8245338112 (7863 MB)
...
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xd000-0x=
d01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 at device 0.0 o=
n pci4
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:11:22:33:44:51
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xc000-0x=
c01f mem 0xf7300000-0xf737ffff,0xf7380000-0xf7383fff irq 19 at device 0.0 o=
n pci5
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:11:22:33:44:52
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci6: <ACPI PCI bus> on pcib6
igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xb000-0x=
b01f mem 0xf7200000-0xf727ffff,0xf7280000-0xf7283fff irq 18 at device 0.0 o=
n pci6
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:11:22:33:44:53
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci7: <ACPI PCI bus> on pcib7
igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xa000-0x=
a01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 19 at device 0.0 o=
n pci7
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 00:11:22:33:44:54
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
...
#> dmidecode
...
System Information
Manufacturer: Supermicro
Product Name: X10SLM+-LN4F
...
#> pciconf -lv
...
igb0@pci0:4:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808=
6 rev=3D0x03 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D 'I210 Gigabit Network Connection'
class =3D network
subclass =3D ethernet
igb1@pci0:5:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808=
6 rev=3D0x03 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D 'I210 Gigabit Network Connection'
class =3D network
subclass =3D ethernet
igb2@pci0:6:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808=
6 rev=3D0x03 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D 'I210 Gigabit Network Connection'
class =3D network
subclass =3D ethernet
igb3@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808=
6 rev=3D0x03 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D 'I210 Gigabit Network Connection'
class =3D network
subclass =3D ethernet
...
=2D --=20
Zeus V. Panchenko jid:zeus@im.ibs.dn.ua
IT Dpt., I.B.S. LLC GMT+2 (EET)
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iEYEARECAAYFAlWvdkEACgkQr3jpPg/3oyraCwCg8OZFNErBWJXmjSHLm+Qm8ruL
IdcAoLG5kxdQpRjv1lPGN3UO27vFY7+H
=3DahV7
=2D----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150722135353.70269>